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PREFACE 


This  report  describes  a  cognitive  task  analysis  methodology  developed  under  the  Basic  Job 
Skills  Research  Program.  The  work  described  was  conducted  by  the  Job  Structures  Branch  of  the 
Manpower  and  Personnel  Research  Division  of  the  Human  Resources  Directorate,  Armstrong 
Laboratory.  It  covers  the  period  from  January,  1985  through  November,  1990. 

We  would  like  to  thank  the  aircraft  maintenance  technicians  assigned  to  Air  Combat 
Command  who  contributed  to  the  studies  reported  here,  and  particularly  the  expert  technicians 
assigned  to  the  laboratory  to  work  on  this  project:  TSgt  Dennis  Collins,  MSgt  Mark  Gallaway, 
and  MSgt  Ron  Kane.  This  work  could  not  have  been  accomplished  without  their  dedicated 
support. 
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SUMMARY 


Developing  effective  instruction  for  complex  problem-solving  tasks  requires  analysis  of  the 
cognitive  processes  and  structures  that  contribute  to  task  performance.  This  report  describes  the 
data  collection  procedures  associated  with  a  cognitive  task  analysis  technique  known  as  the 
“PART’  methodology.  The  methodology  is  being  developed  under  the  Basic  Job  Skills  (BJS) 
program  and  constitutes  one  component  of  an  integrated  technology  for  developing  and  delivering 
training  of  cognitively  complex  tasks.  The  data  collection  procedures  can  be  considered  an 
extension  of  existing  task  analysis  techniques  and  are  based  on  studies  of  over  200  Air  Force 
technicians  in  aircraft  maintenance  specialties  whose  primary  task  is  troubleshooting.  The 
procedures  derived  from  these  studies  impose  a  structure  on  the  knowledge  acquisition  task 
which  captures  the  cognitive  as  well  as  the  behavioral  components  of  troubleshooting  skill.  The 
structured  interview  approach  yields  data  that  allow  qualitative  comparisons  of  problem-solving 
performances  within  and  across  technical  skill  level.  Such  analyses  have  informed  instruction 
developed  under  the  BJS  program  by  revealing  the  developmental  course  of  skill  acquisition  and 
the  components  of  expertise  which  are  the  training  targets.  More  recent  analyses  have  identified 
skill  and  knowledge  commonalities  across  maintenance  specialties  and  are  informing  training 
designed  to  facilitate  knowledge  transfer.  A  future  goal  of  the  BJS  program  is  to  examine  the 
generality  of  the  PARI  methodology  and  the  extent  to  which  it  can  be  applied  to  problem-solving 
tasks  in  nonmaintenance  domains. 
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A  PROCEDURAL  GUIDE  TO  COGNITIVE  TASK  ANALYSIS: 
THE  PARI  METHODOLOGY 


I.  INTRODUCTION 

The  purpose  of  this  document  is  to  provide  a  procedural  guide  to  the  use  of  a  cognitive  task 
analysis  technique  known  as  the  PARI1  methodology.  This  structured,  thinking-aloud  dialogue 
approach  was  developed  as  part  of  the  Basic  Job  Skills  (BJS)  Research  Program  being  carried  out 
at  the  Air  Force  Armstrong  Laboratory.  Although  the  PARI  methodology  incorporates  features 
of  several  existing  task  analysis  techniques,  its  design  was  influenced  by  the  specific  research 
needs  of  the  BJS  program  to  investigate  complex  problem  solving  expertise  in  real  world  work 
environments.  Section  I  of  the  guide  examines  these  requirements  as  they  relate  to  the  program's 
goals  and  the  shaping  of  the  PARI  technique.  A  discussion  of  alternative  task  analysis  procedures 
follows  in  Section  II.  Section  m  provides  a  detailed  description  of  the  PARI  procedure  and 
examines  how  its  features  address  particular  BJS  research  requirements.  Finally,  Section  IV 
discusses  the  broader  uses  of  the  PARI  procedure  as  a  research  tool  and  as  a  general  aid  to 
instructional  design  and  skill  assessment. 

BJS  Project  Overview 
Air  Force  Need  and  Research  Response 

The  BJS  Program  is  a  large-scale  research  effort  directed  at  examining  the  cognitive  skills 
that  allow  individuals  to  interact  adaptively  with  technologically  complex  systems.  The  need  to 
understand  and  foster  advanced  technical  skills  has  been  made  more  urgent  as  a  result  of  two 
parallel  developments.  First,  as  in  most  technical  domains,  the  complexity  of  aerospace  systems 
and  equipment  in  the  Air  Force  is  continuing  to  increase  at  a  phenomenal  rate.  This  development 
poses  an  ever-growing  challenge  to  Air  Force  personnel  as  they  attempt  to  master  their  jobs  by 


1  PARI:  Precursor,  (or  reason  for  Action),  Action,  Result,  and  Interpretation  (of  result)  are 
problem-solving  components  of  interest  in  diagnostic  tasks. 
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acquiring  adequate  knowledge  of  equipment  systems  and  repair  experience  during  the  relatively 
short  period  of  their  enlistment.  At  the  same  time,  and  in  response  to  demands  for  greater 
versatility  among  technical  personnel,  the  Air  Force  has  instituted  a  program  (the  Rivet 
Workforce  initiative)  to  restructure  the  aircraft  maintenance  workforce.  This  program  stipulates 
that  airmen  operating  and  maintaining  technical  equipment  are  to  become  less  specialized  and 
demonstrate  proficiency  in  several  related  job  specialties. 

While  the  first  development  demands  the  acquisition  of  in-depth  system  knowledge  under 
•  considerable  time  constraints,  the  second  development  requires  breadth  and  adaptiveness  of 
knowledge  and  skill  across  multiple  systems.  The  BJS  program  is  a  response  to  this  twofold 
need.  The  goal  is  to  develop  an  integrated  skill  analysis/instructional  development  technology 
that  promotes  both  depth  and  breadth  of  proficiency  in  the  maintenance  of  highly  complex 
systems. 

As  the  skill  analysis  component  of  the  technology,  the  PARI  technique  (which  is  the  focus 
of  this  report)  is  intended  to  allow  both  a  broad  and  deep  examination  of  problem  solving 
expertise.  Toward  this  end,  the  approach  has  evolved  into  a  structured  interview  which  occurs  as 
experts  pose  problems  to  each  other  under  realistic  task  performance  conditions.  A  task  analyst 
uses  standardized  interview  questions  to  systematically  probe  the  experts  during  and  after  solution 
generation.  After  the  solution,  the  experts  are  asked  to  elaborate  the  solutions  they  have  just 
generated  in  a  series  of  rehash  sessions.  Here  they  are  explicitly  asked  to  address  the  factors 
considered  (or  reasons  for)  the  problem  solving  decisions  they  made  during  the  solution  process. 
The  PARI  procedures  thus  yield  fine-grained  (but  systematized)  protocol  data  that  capture  both 
the  solution  steps  and  the  supporting  reasons  to  complex  problems.  From  detailed  protocol  data, 
both  precise  targets  for  in-depth  instruction  as  well  as  broad  skill  commonalitites  across  domains 
can  be  identified.  PARI  output  is  then  incorporated  into  curricula  for  learning  environments  that 
are  designed  to  accelerate  skill  development  as  well  as  foster  flexible,  adaptive  knowledge  bases, 
thereby  addressing  the  two  pressing  Air  Force  needs  described  above. 
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Cognitive  Task  Analysis  Methods  to  Represent  Adaptive  Problem  Solving 


Adaptive  Expertise.  A  focus  on  adaptive  expertise  can  be  contrasted  with  skill  analysis  and 
instruction  directed  at  developing  routinized  expertise.  In  the  case  of  the  latter,  trainees  are 
taught  to  perform  specific  procedures  in  a  highly  efficient  manner  in  the  recognizable  situations  of 
stable  task  environments  (Hatano  &  Inagaki,  1984).  Although  perfectly  appropriate  for  some 
tasks,  such  training  does  not  equip  a  trainee  to  respond  effectively  to  situations  that  are  not 
explicitly  addressed  by  the  training.  In  today's  technologically  sophisticated  workcenters,  novel 
situations  are  frequently  encountered,  thus  making  adaptive  problem  solving  a  distinguishing 
hallmark  of  modem  day  technical  skill.  To  perform  well  in  dynamic,  unstable  task  environments, 
the  performer  needs  flexible  knowledge  and  skill  that  are  adaptable  to  novel  problem  situations. 
This  implies  an  understanding  of  a  job  that  goes  far  beyond  knowledge  of  rote  procedures  for 
operating  and  maintaining  a  piece  of  equipment  and  includes  knowledge  of  why  a  set  of 
procedures  is  appropriate  and  effective  for  a  given  task.  Conceptual  support  knowledge  of  this 
form  empowers  the  human  in  dynamic,  variable  task  environments  by  giving  meaning  to  the  steps 
of  a  procedure,  by  enabling  the  invention  of  new  procedures,  and  by  assisting  in  the 
reconstruction  of  forgotten  procedures.  Making  the  reasons  for  task  performance  explicit  (i.e., 
specifying  this  conceptual  support  knowledge)  is  therefore  a  central  goal  of  the  PARI  approach. 
This  goal  is  being  accomplished  via  several  task  analysis  features. 

Situated  Problem  Solving  and  the  PARI  Procedure.  The  PARI  procedures  revolve  around 
situated  problem-solving  sessions  where  experts  deploy  knowledge  in  response  to  particular 
problem  contexts  and  task  demands.  As  they  seek  solutions,  experts  are  probed  for  the  reasons 
behind  the  actions  they  elect  to  take  and  for  their  interpretations  of  the  results  of  their  actions.  In 
this  way,  the  reasoning  processes  that  are  responsible  for  knowledge  deployment  are  made 
apparent.  The  probes  are  part  of  a  structured  interview  designed  to  reveal  knowledge  and  skill 
in  the  context  of  their  use.  This  approach  contrasts  with  decontextualized  task  analysis  interviews 
where  knowledge  is  abstracted  and  detached  from  the  problem-solving  conditions  under  which  it 
is  normally  applied.  In  the  PARI  interview,  the  full  functional  context  of  the  domain  is 
experienced  so  that  the  various  intended  uses  of  particular  skills  and  knowledge  structures  (i.e., 


3 


the  reasons  behind  the  procedures)  are  made  explicit  for  teaching  (Glaser  et  al.,  1985;  Gott, 

1989). 

Authentic  performance  contexts  are  achieved  in  a  PARI  interview  through  the  dyadic 
interactions  of  pairs  of  experts.2  One  expert  poses  a  problem  to  a  second  expert  who  is  naive 
with  respect  to  the  problem's  source.  The  second  expert  generates  a  solution,  step  by  step,  with 
the  first  expert  providing  the  result  of  each  step.  The  problem  solving  experience  is  thus  a  close 
approximation  of  the  dynamic  real  world  where  causes  of  malfunctions  and  results  of  actions  are 
not  known  in  advance.  Not  surprisingly,  experts  solve  problems  much  differently  when  they 
know  the  source  of  the  problem.  When  they  are  naive  to  the  problem  source,  they  consider  a 
much  wider  range  of  hypotheses  and  correspondingly  search  and  access  richer  knowledge 
structures.  As  a  consequence,  an  analysis  of  naive  problem-solving  is  more  fruitful  in  revealing  all 
relevant  knowledge  bases,  search  procedures,  and  strategic  deployment  processes.  The  expert 
dyad  represents  a  unique  feature  of  the  PARI  methodology  and  serves  to  ground  all  knowledge 
elicitation  firmly  in  authentic  contexts. 

Another  PARI  feature  worthy  of  note  is  that  the  task  analyst  interviews  multiple  experts 
and  gathers  data  on  a  representative  sample  of  domain  problems.  The  solutions  that  are 
produced  are  thus  more  likely  to  represent  the  full  range  of  expert  problem-solving  performances 
and  the  collective  domain  knowledge  of  expert  practitioners.  Situated,  problem-based  learning 
environments  can  flow  fairly  easily  from  the  products  of  a  dyadic,  representative  task  analysis 
such  as  this.  Additional  methodological  features  ensure  that  PARI  interview  data  are  both  reliable 
and  standardized. 

Standardization  and  Codification  of  PARI  Procedures.  Standardizing  and  codifying  any  task 
analysis  methodology  is  important  to  its  continued  utility,  but  the  reasons  are  particularly 
compelling  for  codifying  PARI  procedures.  The  rationale  is  tied  to  the  BJS  research  goal  to 
represent  both  the  depth  and  breadth  of  technical  proficiency.  In  order  to  investigate  skill  and 

2  Allan  Collins,  BBN  and  Northwestern  University,  provided  the  initial  inspiration  for  this  dyadic, 
in  situ  problem  solving  approach. 
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knowledge  commonalities  and  associated  processes  of  transfer,  a  comparison  of  task  analysis 
results  across  job  specialties  is  necessary.  Within-job  comparisons  of  expert  and  novice  problem 
solving  capabilities  are  also  needed  to  examine  the  development  of  domain  knowledge  and 
problem-solving  skill.  Comparative  analyses  require  data  having  a  standard  form.  Imposing  a 
structure  on  the  PARI  problem-solving  sessions  with  the  use  of  standard  probes  ensures  uniform 
data  structures  across  different  task  analyses,  thereby  facilitating  these  types  of  comparisons. 

A  related  methodological  issue  concerns  the  implementation  demands  imposed  on  the  task 
analysts  who  use  the  procedures.  To  understand  these  demands,  it  should  be  noted  that  the 
primary  emphasis  of  the  BJS  program  is  to  provide  the  Air  Force  with  a  technology  for  skill 
analysis  and  training  development  in  technical  domains.  Since  the  purpose  of  such  a  technology  is 
to  allow  the  development  of  instruction  by  Air  Force  personnel  who  may  not  be  cognitive 
psychologists,  instructional  design  specialists,  or  even  subject-matter  experts,  the  technology 
should  provide  a  principled  and  implementable  method  for  both  task  analysis  and  instructional 
development  by  nonscientists.  This  practical  concern  underscores  the  need  for  well-codified 
procedures  for  (a)  conducting  task  analysis  studies,  (b)  compiling  and  interpreting  resultant  data, 
and  (c)  designing  and  developing  training  based  on  the  findings.  (This  general  issue  of 
methodological  codification  will  be  discussed  in  detail  in  Section  HI  of  this  guide.) 

While  the  research  requirements  of  the  BJS  program  have  significantly  influenced  the 
particular  form  of  our  task  analysis  procedures,  the  goal  of  capturing  the  conceptual  knowledge 
underlying  complex  task  performances  (or  reasons  for  procedural  sequences)  reflects  a  common 
objective  of  many  cognitive  analysis  approaches.  In  the  next  section,  we  describe  the  cognitive 
components  of  skill  that  are  typically  targeted  by  such  approaches.  More  specifically,  we  describe 
the  components  of  problem  solving  expertise  that  have  been  consistently  derived  in  BJS  empirical 
studies  and  by  research  in  cognitive  psychology  in  general.  The  intent  is  to  characterize  more 
concretely  the  multifaceted  knowledge  structures  that  the  PARI  procedure  is  designed  to  capture. 
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Cognitive  Components  of  Complex  Problem  Solving 


Ill-Structured  Problem  Solving 

The  diagnosis  of  faults  in  complex  aerospace  systems  is  an  activity  which  can  be 
characterized  as  ill-structured  problem  solving:  there  is  no  well-defined  algorithm  for  problem 
solution,  the  goal  structure  of  the  solution  may  be  poorly  defined,  and/or  the  criteria  for 
evaluating  solution  acceptability  may  be  weak  (Newell,  1969).  To  date  the  study  of  ill-structured 
problem  solving  has  yielded  functional  distinctions  between  different  knowledge  structures  that 
contribute  to  skilled  performance.  These  structures  include  procedural  and  declarative  knowledge 
(knowing  how  as  well  as  knowing  that),  and  strategic  decision  factors  or  processes  that  are 
responsible  for  knowledge  deployment  and  also  serve  an  executive  control  function  in  problem 
solving.  Gott  (1989)  summarizes  the  role  of  each  of  these  cognitive  components  in  solving 
problems  involving  the  interaction  of  a  human  with  a  complex  system: 

Whether  operating  a  word  processor  or  diagnosing  a  faulty  engine,  the 
human  performer  is  required  to  select  and  execute  procedures  to 
interact  with  an  object  to  achieve  a  set  of  goals.  The  knowledge  and 
processes  that  constitute  that  performance  are  (a)  procedural  (or  how- 
to-do-it)  knowledge,  (b)  declarative  (domain)  knowledge  of  the  object 
(often  called  system  or  device  knowledge),  and  (c)  strategic  (or  how- 
to-decide-what-to-do-and-when)  knowledge.  With  this  decomposition 
it  is  assumed  that  procedural  and  device  knowledge  are  organized  and 
deployed  by  mechanisms  such  as  the  goals,  plans,  and  decision  rules 
that  comprise  strategic  knowledge  (Gott  &  Pokomy,  1987).  This 
deployment  capability  serves  a  control  function  to  enable  what  can  be 
called  dynamic,  opportunistic  reasoning.  Ideally,  this  results  in  optimal 
solutions  crafted  in  response  to  particular  situations  by  applying  just  the 
right  piece  of  knowledge  at  just  the  right  time.  (p.  100) 
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A  model  illustrating  these  interactions  is  depicted  in  Figure  1 . 


Figure  1.  BJS  Cognitive  Architecture. 

Coordination  of  Knowledge  Structures 

Experts  appear  to  coordinate  these  multiple  types  of  knowledge  quite  efficiently,  generating 
plans  and  goal  structures  that  deploy  device  models  and  procedures  that  virtually  ensure  problem 
solution.  The  goal-oriented  nature  of  experts'  problem-solving  activity  involves  the  consideration 
of  multiple  alternative  solution  paths  and  the  judicious  selection  of  alternative  paths  to  pursue. 
Moreover,  the  solution  search  is  (optimally)  updated  as  new  evidence  about  the  probable  source 
of  the  problem  is  uncovered.  The  lack  of  well-defined  algorithms  (as  noted  above  by  Newell) 
arises  partly  from  the  number  of  alternative  methods  available  for  solving  such  problems  and  from 
the  fact  that  the  expert's  choice  among  these  alternatives  depends  on  information  actively  obtained 
through  on-line  hypothesis  testing.  Further,  experts  do  not  appear  to  consider  every  one  of  the 
virtually  unlimited  number  of  solution  paths  available  in  a  complex  problem  (Glaser  et  al.,  1985). 
Rather,  they  bring  to  bear  a  great  deal  of  acquired  knowledge  to  construct  and  constrain  the 
problem/solution  space.  The  development  of  various  types  of  acquired  knowledge  and  their  roles 
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in  problem  solving  will  clarify  the  reasons  for  the  functional  distinctions  researchers  have  made 
between  procedural,  system,  and  strategic  knowledge  sources. 

Procedural  Knowledge.  The  most  readily  observable  aspects  of  task  performance  are  the 
procedures  that  are  executed  in  carrying  out  the  task.  Taking  measurements,  swapping 
components,  and  running  diagnostic  tests  are  examples  of  procedures  commonly  executed  in  tasks 
like  electronic  troubleshooting.  While  knowledge  of  how  to  execute  these  procedures  is  a 
prerequisite  for  troubleshooting  expertise,  it  alone  cannot  account  for  such  expertise.  A  variety  of 
expert-novice  studies  have  shown  that  when  technicians  are  tested  on  basic  troubleshooting 
operations  outside  the  context  of  solving  actual  problems,  differences  in  skilled  and  unskilled 
subjects'  performance  are  not  substantial  (Gitomer,  1984;  Glaser  et  al.,  1985;  Soloway,  1986).  It 
is  experts'  ability  to  select  procedures  in  such  a  way  as  to  optimize  the  efficiency  of 
troubleshooting  that  markedly  distinguishes  them  from  unskilled  technicians.  This  ability  in  turn 
appears  to  depend  heavily  on  knowledge  of  how  the  system  or  device  works. 

Declarative  fSvstem)  Knowledge.  Declarative  (or  how-it-works)  knowledge  appears  to 
provide  the  basis  for  the  adaptiveness  that  characterizes  technical  expertise  in  domains  dominated 
by  ill-structured  problems.  Knowing  why  procedures  work,  for  instance,  allows  known 
procedures  to  be  adapted  to  novel  situations  and  to  be  reconstructed  if  forgotten.  Much  of  the 
research  directed  at  examining  how  declarative  knowledge  enables  this  flexible  use  of  procedural 
knowledge  has  been  concentrated  in  the  domain  of  complex  system  operation  and  maintenance. 
This  work  has  suggested  a  close  association  between  one's  understanding  of  why  procedures 
work  and  knowledge  of  how  the  system  itself  works. 

In  experiments  reported  by  Kieras  and  Bovair  (1984),  subjects  were  asked  to  operate  a 
simple  control  panel  device.  One  group  learned  a  set  of  operating  procedures  bv  rote  while  the 
second  group  was  provided  information  about  the  device's  topology  and  internal  structure  prior  to 
receiving  training  on  procedures  identical  to  that  of  the  first  group.  The  second  group  was  thus 
provided  with  a  "device  model"  and  appeared  to  use  this  model  for  learning  the  procedures  and 
for  performing  the  task:  they  learned  the  procedures  faster,  retained  them  more  accurately, 
executed  them  faster,  and  simplified  the  inefficient  procedures  more  often  than  the  rote  group.  In 
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a  second  study,  subjects  who  either  had  no  training  at  all  or  had  received  device  topology 
instruction  were  asked  to  operate  the  device;  neither  group  received  explicit  training  in  operating 
procedures.  In  contrast  to  subjects  in  the  no-training  group  (who  learned  how  to  operate  the 
device  by  trial  and  error),  those  in  the  device  model  group  were  able  to  operate  the  device  almost 
immediately. 

This  study  lends  striking  support  to  the  view  that  for  domains  where  problem-solving 
activity  is  directed  at  the  operation  and  maintenance  of  equipment  systems,  the  technician's 
understanding  of  the  system  constitutes  critical  content  of  the  declarative  knowledge  base.  Such 
knowledge  includes  facts  about  the  system's  internal  structure,  the  functions  of  various  system 
components,  how  components  operate  and  interact;  in  short,  how  the  system  works.  As  problems 
are  solved,  relevant  facts  are  selected  from  this  base  and  organized  to  form  a  device  model  that 
can  be  elaborated  as  new  facts  about  the  device's  current  behavior  are  discovered,  and  mentally 
manipulated  to  formulate  hypotheses  and  determine  the  next  appropriate  step  in  the  solution. 

The  contribution  of  system  understanding  or  device  knowledge  to  troubleshooting  expertise 
is  further  documented  by  other  studies  comparing  the  depth  and  quality  of  this  knowledge  in 
expert  and  novice  technicians  (Tenney  &  Kurland,  1988;  Gitomer,  1984;  Glaser  et  al.,  1985; 
Means  &  Gott,  1988.)  In  general  this  body  of  work  supports  the  conclusions  summarized  below 
regarding  expert  and  novice  system  knowledge,  strategic  knowledge,  and  associated  problem 
solving. 

There  are  striking  differences  in  the  quality  of  novice  vs  expert  device  representations  as 
well  as  the  ways  that  experts  and  novices  use  their  knowledge  of  the  system  to  construct  and 
constrain  the  problem  space.  Whereas  experts  are  able  to  relate  symptoms  to  possible  inoperative 
system  functions,  novices  tend  to  relate  symptoms  to  equipment  components  that  are  located  in 
the  immediate  physical  vicinity  of  the  initial  symptoms.  This  suggests  a  deep  vs  surface  structure 
difference  in  expert  and  novice  device  models.  The  deep  structure  models  of  experts  allow  an 
initial  problem  representation  that  incorporates  functional  areas  of  the  equipment  that  could  be 
responsible  for  the  observed  symptoms.  Experts  then  solve  the  problems  by  systematically  and 
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efficiently  eliminating  each  of  these  areas  as  the  probable  source.  Novice  solutions  on  the  other 
hand,  tend  to  be  characterized  by  an  initial  focus  on  components  that  are  physically  near  the 
component  displaying  the  symptom,  regardless  of  the  functional  likelihood  that  proximal 
components  could  be  the  cause.  This  is  consistent  with  the  superficial  model  of  the  system  they 
presumably  possess.  If  such  localized  options  fail  to  isolate  the  fault,  subsequent  searches  by 
novices  appear  to  be  random  (Gott,  1989;  personal  communication).  Due  to  their  lack  of  system 
understanding,  novices  are  apparently  unable  to  think  of  the  equipment  in  terms  of  functional  units 
which  would  help  them  generate  effective  plans  for  an  efficient  investigation. 

Strategic  Knowledge.  Differences  in  strategic  decision  making  by  expert  and  novice 
troubleshooters  have  also  been  established  in  comparative  studies  of  problem  solvers  (Gitomer, 
1984;  Glaser  et  al.,  1985).  Strategic  knowledge  is  used  to  select  and  deploy  relevant  system  and 
procedural  knowledge  for  the  purposes  of  establishing  goals,  planning  solutions,  evaluating  the 
outcomes  of  steps  in  the  solution,  and  monitoring  the  progress  of  the  solution.  The  compiled 
empirical  evidence  suggests  that  differences  in  strategic  knowledge  are  related  both  to  the  deeper 
and  more  elaborated  structures  of  the  expert's  system  knowledge  and  to  problem  solving 
strategies  which  direct  the  search  for  solution-relevant  information. 

The  expert's  greater  knowledge  of  the  system  increases  the  number  and  quality  of  alternative 
hypotheses  considered,  while  facility  with  basic  troubleshooting  operations  allows  these 
hypotheses  to  be  tested  in  a  variety  of  ways.  The  expert  thus  has  a  number  of  alternative  solution 
paths  available  and  the  resulting  procedural  flexibility  permits  the  construction  and  on-line 
refinement  of  a  troubleshooting  plan  and  associated  operations  that  are  specifically  adapted  to  the 
problem  being  solved.  In  sum,  experts  are  able  to  engage  in  dynamic,  opportunistic  reasoning 
because  they  have  richer  system  knowledge  bases  and  a  wider  range  of  tools  (operations)  for 
implementing  their  plans. 

This  interpretation  acknowledges  the  dependence  of  strategic  processing  on  the  content  and 
organization  of  specific  device  and  procedural  knowledge  structures  (see  also  Chi,  Glaser,  & 

Rees,  1982;  Greeno,  1980).  This  is  in  contrast  to  a  view  of  the  strategic  component  of  expertise 
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as  domain-independent  or  general  (weak)  problem-solving  heuristics.  A  result  reported  by  Glaser 
et  al.  (1985)  illustrates  this  distinction  and  supports  the  "knowledge-based"  view  of  expert 
strategic  decision  making.  These  investigators  examined  the  generation  of  plans  by  skilled  and 
less  skilled  avionics  technicians  in  a  verbal  troubleshooting  task.  While  the  two  groups  did  not 
differ  with  respect  to  the  frequency  of  plan  generation,  the  quality  of  these  plans  varied  in 
predictable  ways: 

Both  higher  and  lower  skill  subjects  were  planful.  The  differences  arise 
only  when  the  utility  of  the  plan  for  solving  the  problem  is  considered. 

Put  another  way,  the  less  skilled  subjects  have  equivalent  "weak" 
methods  to  their  more  skilled  colleagues  but  are  way  behind  on 
stronger  problem-solving  methods,  methods  particularized  to  the 
specific  technical  domain  (p.  283). 

Thus,  although  domain-independent  strategies  (weak  methods)  may  be  used  by  both  experts  and 
novices,  it  is  the  experts  who  deploy  particularized,  strong  methods. 

It  is  often  strategic  knowledge  that  is  overlooked  in  standard  textbook  presentations  of 
course  material.  For  example,  in  examining  instructional  materials  for  geometry  courses,  Greeno 
(1978)  observed  that  of  the  three  components  necessary  for  problem  solving  in  geometry, 
strategic  knowledge  was  neither  explicitly  taught  by  teachers  nor  directly  treated  in  textbooks. 
Correspondingly,  Chi,  Glaser,  and  Rees  (1982)  note  that  when  problem  solving  strategies  are 
studied  in  the  context  of  relatively  knowledge-free  tasks  (e.g.,  in  studies  using  puzzle  problems), 
limited  insights  are  gained  into  principles  that  guide  the  search  of  a  large  knowledge  base.  In  both 
cases,  the  relationship  between  knowledge  of  the  domain  and  the  strategies  used  to  solve 
problems  fails  to  become  established.  In  sum,  the  strategic  knowledge  that  enables  planning  and 
decision  making  by  expert  problem  solvers  may  be  overlooked  when  domain  knowledge  is 
analyzed  outside  the  context  of  its  intended  use,  i.e.,  decontextualized  from  actual  problem 
conditions.  The  importance  of  capturing  the  procedural,  declarative,  and  strategic  components 
become  clearer  as  the  design  of  instruction  is  contemplated. 
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The  Cognitive  Analysis  of  Expertise  as  Input  to  Instruction 
Cognitive  vs  Behavioral  Task  Analysis 

Cognitive  task  analyses  were  first  used  to  inform  instructional  development  during  the  mid 
seventies  (Greeno,  1976;  Resnick,  1976)  as  psychological  theories  of  cognitive  processing  became 
sufficiently  mature  to  yield  specific  analytic  procedures.  Prior  to  that  time,  the  dominant  approach 
to  task  analysis  could  be  classified  as  a  "behavioral-rational"  method  (Gott,  1986;  Gagne,  1974; 
1977).  A  brief  comparison  of  these  approaches  may  make  clearer  the  utility  of  a  cognitive 
analysis. 

One  distinguishing  feature  of  the  two  approaches  is  their  differing  orientations  to 
instruction.  Whereas  the  purpose  of  a  behavioral-rational  task  analysis  is  primarily  to  specify  the 
component  steps  of  the  observable  behaviors  as  targets  of  instruction,  e.g.,  how  to  operate  a 
voltmeter  (Anderson  &  Faust,  1973),  a  cognitive  analysis  is  targeted  at  the  underlying 
psychological  processes  and  knowledge  structures  that  are  required  to  produce  the  correct  overt 
behaviors  at  the  appropriate  time,  e.g.,  understanding  what  a  voltmeter  does,  when  its  use  is 
indicated,  what  its  resultant  data  reveal,  and  so  forth  (Greeno,  1976).  Thus,  while  instruction 
based  on  behavioral  analyses  may  emphasize  the  specific  behavioral  steps  required  to  perform  a 
task,  instruction  based  on  a  cognitive  analysis  also  teaches  the  psychological  underpinnings  of 
these  behavioral  steps,  or  the  conceptual  support  knowledge  that  both  explains  the  solution  and 
fosters  adaptiveness,  as  discussed  earlier.  This  difference  in  instructional  orientation  is  reflected  in 
the  content  of  the  knowledge  bases  produced  by  the  two  types  of  task  analysis. 

Traditionally,  emphasis  has  been  placed  on  the  identification  of  observable  behaviors  as 
instructional  targets;  therefore,  procedural  (or  how-to-do-it)  knowledge  is  readily  captured  in 
behavioral-rational  analyses.  However,  other  equally  important  forms  of  knowledge  used  by 
experts  in  complex  problem  solving  tasks  may  be  ignored.  For  example,  declarative  (or  factual) 
knowledge  and  strategic  knowledge  may  have  no  outward  performance  manifestations  and  are 
thus  difficult  to  capture  with  more  traditional  methods.  The  benefits  of  cognitive  models  that 
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specify  such  unobservable  psychological  processes  and  knowledge  structures  are  supported  by  a 
growing  body  of  empirical  data.  The  evidence  shows  that  on  knowledge-rich  tasks,  the  typical 
novice  deficiencies  are  not  procedural  (or  how-to-do-it  knowledge),  but  rather  tend  to  be  gaps  in 
declarative  knowledge  bases  and  uncertainties  about  when  to  deploy  specific  pieces  of  knowledge 
(Glaser  et  al.,  1985;  Soloway,  1986;  Gitomer,  1984). 

Because  cognitive  analyses  examine  all  relevant  knowledge  sources  rather  than  focusing 
exclusively  on  the  single  source  that  is  most  directly  tied  to  observable  behaviors  (i.e.,  procedural 
knowledge),  they  yield  more  detailed  representations  of  tasks  to  be  used  as  instructional  targets 
(Gott,  1989).  In  these  representations,  the  goals  to  which  procedural  knowledge  applies  and  the 
strategic  processes  that  are  responsible  for  the  organization,  coherence,  and  general  execution  of 
the  performance  are  clearly  established.  Knowledge  is  not  only  directly  tied  to  its  uses  in  the  real 
world,  but  knowledge  that  is  typically  tacit  (including  goals,  strategies,  and  assumptions)  is  made 
explicit  for  teaching.  Cognitive  analysis  data  as  input  to  instruction  thereby  offers  the  potential 
for  a  more  complete  treatment  of  the  multifaceted  forms  of  knowledge  used  in  complex  problem 
solving.  In  turn,  adaptive  performance  becomes  a  more  realistic  instructional  goal. 

Pedagogical  design  can  be  further  informed  by  the  results  of  cognitive  analyses  that  reveal 
developmental  aspects  of  the  skill  acquisition  process,  specifically,  the  differences  among 
individuals  at  different  proficiency  levels  (Gott,  1986).  The  way  that  knowledge  is  acquired  and 
organized  and  how  skills  are  developed  over  time  can  be  captured  and  used  to  inform  the 
sequencing  of  instructional  events.  In  that  way  students  are  naturally  taken  through  successive 
approximations  of  mature  practice  as  they  learn  how  to  perform  increasingly  complex  tasks  (Gott, 
1989). 

We  have  argued  thus  far  that  training  targeted  at  modem  day  expertise  must  make  explicit 
the  multiple  types  of  knowledge  that  contribute  to  overt  skill.  Cognitive  task  analyses  can  be 
viewed  as  extensions  of  traditional  behavioral  techniques  in  this  regard.  The  following  discussion 
examines  instructional  systems  that  have  been  informed  by  cognitive  models  of  skill  acquisition 
and  performance. 
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Instructional  Trends 


In  their  recent  reviews  of  cognitive  theory-based  instructional  systems,  both  Glaser  and 
Bassok  (1989)  and  Gott  (1989)  note  the  contribution  of  cognitive  task  analysis  methods  to  the 
development  of  advanced  instructional  systems.  They  point  out,  however,  that  few  instructional 
designers  have  attempted  to  integrate  the  multiple  knowledge  sources  that  contribute  to  skilled 
performance  or  to  make  these  knowledge  structures  explicit  to  the  student.  Instead,  instruction 
tends  to  focus  on  the  acquisition  of  a  single  (or  sometimes  dual)  skill  component,  (e.g.,  procedural 
skill  and  goal  structure  knowledge  in  Anderson's  LISP  tutor  [Anderson,  Boyle,  &  Reiser,  1985; 
Anderson  &  Reiser,  1985]  or  strategic  planning  and  self-monitoring  in  the  reading  comprehension 
program  of  Brown  and  Palincsar  [1984;  1988]).  Next  we  consider  some  of  the  limitations  of 
instructional  systems  that  represent  this  more  common  single  or  dual  component  instructional 
approach. 

Researchers'  attempts  to  represent  domain  knowledge  in  a  form  that  would  yield  executable 
models  of  task  performance  provided  the  basis  for  several  systems  representing  the  cognitive 
approach  to  instruction  (O'Shea,  1979a;  1979b;  Clancey,  1979a;  1979b;  Sleeman  &  Smith,  1981; 
Anderson,  et  al.,  1985;  Anderson  &  Reiser,  1985).  The  cognitive  performance  models  underlying 
these  "tutors"  incorporated  production  rules  as  the  computational  structure  in  which  knowledge 
was  represented.  Production  rules  can  be  described  as  condition-action  pairs:  they  state  the 
conditions  under  which  a  particular  action  is  to  be  taken,  as  well  as  the  action  itself.  Thus,  the 
model  can  be  said  to  represent  knowledge  of  both  "how-to-do-it"  (procedural  knowledge)  and 
"when-to-do-it"  (strategic  knowledge). 

Although  quite  powerful  in  terms  of  their  ability  to  solve  problems,  production  systems  have 
difficulty  in  providing  a  sound  theoretical  basis  for  humans'  ability  to  solve  ill-structured  problems. 
First  of  all,  in  domains  characterized  by  such  problems,  the  number  of  rules  required  to  represent 
all  goal  states  and  their  associated  actions  would  be  virtually  unlimited.  Given  the  assumption  of 
such  tutors  that  production  rules  are  the  units  of  knowledge  to  be  acquired,  Gott  (1989)  notes 
that  the  student  must  learn  a  large  set  of  independent,  situation-specific  rules  with  limited 
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generalizability,  thus  raising  the  question  of  whether  adaptability  to  unfamiliar  situations  requires 
knowledge  of  a  different  sort.  Further,  Rouse  (1982)  states  a  serious  concern  with  this  type  of 
"procedural  knowledge"  acquisition  in  complex  real  world  domains: 

...  the  proceduralization  approach  is  fundamentally  limited  by  the  fact  that 
one  can  seldom  anticipate  all  of  the  events  that  may  occur  in  a  particular 
system.  Thus,  the  operators  and  maintainers  inevitably  encounter  an  event 
for  which  there  is  no  procedure,  or  a  combination  of  events  for  which  it  is 
not  clear  which  procedure,  if  any,  should  be  used.  In  such  situations, 
proceduralized  training  is  of  no  use  (p.  104). 

In  fact,  Anderson  himself  has  noted  that  the  production-system  approach  is  more  suited  to 
"algorithmically  tractable  domains"  (1987)  and  that  in  some  instances,  "more  generalized 
declarative  knowledge  may  be  desired"  (1988). 

The  limitations  of  instructional  systems  that  require  students  to  learn  large  numbers  of 
procedures  or  rules  by  rote  were  made  clear  in  field  tests  of  GUIDON,  a  medical  diagnosis  tutor 
based  on  an  expert  system  called  MYCIN.  Students  who  participated  in  these  tests  had  difficulty 
understanding  and  remembering  MYCIN's  rules.  Because  MYCIN  did  not  use  or  represent  either 
the  reasoning  strategies  used  by  humans  in  medical  diagnosis  or  knowledge  of  how  the  human 
system  works,  these  knowledge  components  (which  are  required  in  diagnostic  problem  solving  by 
humans)  could  not  be  communicated  to  students  by  GUIDON.  Students  could  not  make  sense  of 
MYCIN’s  production  rules  so  as  to  integrate  them  into  a  useful  body  of  knowledge  (Clancey, 
1984).  Although  MYCIN's  rules  are  a  "machine-efficient"  framework  for  the  representation  of 
knowledge,  they  represent  "compiled"  expertise  which  makes  them  too  obscure  for  students  to 
comprehend  and  retain  (Wenger,  1987). 

More  recent  work  by  Clancey  and  his  colleagues  has  focused  on  the  decompilation  of 
expertise  and  modelling  the  reasoning  strategies  used  in  problems  of  heuristic  classification.  In 
the  newer  systems  resulting  from  this  work  (e.g.,  NEOMYCIN  and  GUIDON2)  reasoning 
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strategies  and  different  types  of  knowledge  (e.g.,  general  principles,  common  world  facts, 
definitional  and  taxonomic  relations)  are  discreetly  represented  so  that  the  diagnostic  problem 
solving  process  can  be  meaningfully  communicated  to  students. 

PARI  Data  as  Input  to  Instruction 

The  PARI  approach  to  instructional  development  has  benefitted  from  the  lessons  of  both 
behavioral  and  early  cognitive  instructional  approaches.  The  methodology  reflects  the  need  to 
capture  the  cognitive  underpinnings  of  complex  task  performance  by  human  problem  solvers,  in 
order  to  identify  the  knowledge  that  is  required  for  human  students  to  understand  the  problem 
solving  process.  The  features  of  the  methodology  outlined  earlier  in  this  section  (e.g.,  situated 
problem  solving  and  dyadic  interaction)  enable  the  specification  of  procedural,  declarative,  and 
strategic  knowledge,  as  well  as  the  coordinated  deployment  of  all  three  knowledge  components 
during  dynamic,  opportunistic  problem  solving. 

The  ultimate  purpose  of  the  PARI  procedure  is  to  provide  models  of  a  range  of 
performances  that  can  then  be  used  to  guide  the  design  of  instructional  systems  which  target  the 
acquisition  and  integration  of  all  three  skill  components.  There  are  of  course  other  task  analytic 
procedures  that  are  directed  at  producing  models  of  performance  for  various  uses.  Some  of  these 
procedures  also  focus  on  underlying  cognitive  processes  and  knowledge  structures  as  a  way  to 
explain  observable  behaviors.  Selected  alternative  methods  will  be  briefly  reviewed  in  the 
following  section  in  order  to  sharpen  the  rationale  for  our  development  of  the  PARI  approach. 
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H.  ALTERNATIVE  TASK  ANALYSIS  APPROACHES 


Instructional  Systems  Design  (ISD) 


Purpose 

Instructional  systems  design  or  the  ISD  approach  is  a  methodology  that  has  been  widely 
used  by  the  Air  Force  since  1972  for  designing  and  developing  Air  Force  training  programs 
(Devries,  Eschenbrenner,  &  Ruck,  1980).  As  a  task  analytic  approach,  the  ISD  procedure  has 
been  described  as  a  general-purpose  tool  suitable  for  the  decomposition  of  a  wide  variety  of  tasks. 
The  tasks  to  be  decomposed  are  determined  by  a  coarse-grained  analysis  of  a  job  which  identifies 
the  duties  of  that  job  (e.g.,  the  job  of  vehicle  mechanic  involves  duties  such  as  adjusting  and 
repairing  brakes,  tuning  engines,  repairing  electrical  systems,  and  so  forth).  Each  of  these  duty 
categories  comprises  a  set  of  tasks  (e.g.,  tuning  engines  might  require  distributor  repair,  plug 
replacement,  carburetor  repair,  etc.).  Task  analysis  or  decomposition  then  involves  two 
components:  (a)  identifying  the  subtasks  that  define  task  performance  and  (b)  specifying  the  skills 
and  knowledge  underlying  each  subtask. 

Procedure/Result 


The  identification  of  subtasks  is  accomplished  through  the  observation  of  the  task  being 
performed  by  a  subject-matter  expert  under  simulated  or  actual  job  performance  conditions.  The 
analyst  lists  the  steps  or  overt  acts  in  the  task,  indicating  exactly  what  is  done  and  how  the  steps 
are  performed,  whether  and  how  any  equipment  is  used,  and  any  decisions  that  are  required 
during  the  procedure.  The  major  steps  identified  in  the  procedure  correspond  to  subtasks  which 
become  the  focus  of  the  next  stage  of  the  analysis.  In  this  stage,  the  declarative  knowledge  and 
physical  and  manipulative  skills  required  to  perform  each  subtask  are  listed.  The  analyst  may  rely 
on  task  observation,  studies  of  job  documentation,  and  the  judgment  of  the  subject  matter  expert 
in  identifying  supporting  skills  and  knowledge.  The  task  analysis  documentation  form  and 
resultant  data  are  illustrated  in  Table  1. 
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Table  1 

ISD  task  analysis  documentation  form  (adapted from  DeVries,  Eschenberenner,  &Ruck,  1980) 


STS/CTS  TASK:  ISOLATE  DEFECTIVE  PARTS  IN/ON  POWER  SUPPLY  CIRCUIT 

BEHAVIOR:  ISOLATE  DEFECTIVE  PARTS  IN/ON  POWER  SUPPLY  CIRCUIT 

CONDITIONS:  GIVEN  MULTIMETER,  0*SC0PE  AND  RAM'S  175  &  176 

STANDARDS:  W1THN  15  MINUTES  EACH 

REFERI 

ENCES  KAMS 175  A  176,  AFRAM  525,  KAO  81 

STEP 

STEP  (SUBTASK/DECISION  QUESTION) 

DECISIONS 

GOTO 

SUPPORTING  SKI  US  A  KNOWLEDGES 

NO. 

YES 

NO 

STEP 

1 

CHECK  FUSE  FOR  CONTINUITY 

WITH  MULTIMETER 

2 

S-USE  OF  MULTIMETER 

K  -  LOCATION  OF  FUSE 

2 

IS  FUSE  CONTINUOUS 

5 

K-  CRITICAL  READING 

ON  MULTIMETER 

3 

CHECK  RECTIFIER  CIRCUITS 

WITH  OSCILLOSCOPE 

4 

S  -  USE  OF  OSCILLOSCOPE 

K  -  RECTIFIER  DIODE  OUTPUTS 

K  -  USE  OF  RAM'S  175  A 176 

4 

CHECK  REGULATOR  CIRCUITS 

WITH  MULTIMETER 

STOP 

S  -USE  OF  MULTIMETER 

K  -  REGULATOR  VOLTAGE  OUTPUTS 
K-  USE  OF  RAM'S  175  A 176 

5 

CHECK  TRANSFORMER  IMTH 
OSCILLOSCOPE 

6 

S  -  USE  OF  OSCILLOSCOPE 

K  -  HIGH  VOLTAGE  SAFETY 
PRECAUTIONS 

K  -  TRANSFORMER  ACTION 

K  -  USE  OF  RAM'S  175  A 176 

6 

CHECK  FILTER  CIRCUITS 

WITH  MULTIMETER 

1 

STOP 

S-USE  OF  MULTIMETER 

K  -  CAPACITIVE  REACTANCE 

K  -  USE  OF  RAM'S  175  A 176 

The  behavior  (or  task)  of  interest  in  this  example  is  to  "isolate  defective  detailed  parts  in/on 
power  supply  circuits."  DeVries  et  al.  (1980)  refer  to  this  behavior  as  a  "variable  sequence, 
equipment  oriented  task,"  and  they  use  a  flowchart  to  diagram  the  decision  sequence  (see  Figure 
2).  Associated  with  each  subtask/decision  question  are  the  supporting  skills  and  knowledge  the 
task  analyst  either  observes  or  infers  from  the  technician's  behavior  and/or  associated  task 
documentation.  The  output  from  an  ISD  analysis  closely  parallels  the  condition-action  rules 
discussed  earlier  as  "how-to-do-it"  or  procedural  knowledge.  The  units  of  knowledge  to  be 
acquired  in  both  cases  are  procedural  rules,  e.g.,  "If  the  goal  is  to  isolate  a  defective  part  in  a 
power  supply  circuit,  then  check  the  fuse  for  continuity  using  a  multimeter;  if  the  fuse  is 
continuous,  check  rectifier  circuits  using  an  oscilloscope;  and  so  forth."  What  is  not  clear  from 
this  type  of  analysis  are  the  reasons  underlying  the  sequence  of  procedures,  e.g.,  why  start  with 
the  fuse  and  then  go  to  the  rectifier  circuits?  More  explicit  information  about  strategic  decision 
factors  and  the  specific  device  models  of  the  power  supply  that  are  being  used  would  be  required 
to  flesh  out  the  underlying  reasons  (conceptual  support  knowledge)  for  this  particular  task 
performance. 
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ISOLATE  DEFECTIVE  DETAILED  PARTS  IN/ON 
POWER  SUPPLY  CIRCUITS 


SAFETY  PRECAUTIONS 
S  -USE  OF  OSCLLOSCOPE 


Figure  2.  ISD  task  analysis  flowchart  (adapted  from  DeVries,  Eschenbrenner,  &  Ruck,  1980). 
Relationship  to  the  PARI  Approach 

There  are  several  possible  reasons  why  device  models  and  strategic  knowledge  are  not 
targeted  more  directly  by  the  ISD  approach.  The  relatively  simple  nature  of  the  procedure  being 
analyzed  in  this  example  may  make  any  deeper  cognitive  analysis  of  the  task  unjustifiable,  i.e.,  not 
"cost  effective"  for  instructional  development  purposes.  In  addition,  the  format  of  the  ISD 
methodology  is  such  that  the  performer  knows  in  advance  the  location  of  the  source  of  the 
problem.  This  advance  knowledge  removes  the  need  for  the  problem  solver  to  investigate 
alternative  hypotheses,  which  in  turn  leaves  "unactivated"  (and  thus  undocumented)  many 
strategic  decision  control  processes  and  alternative  device  models. 

As  a  consequence,  the  ISD  task  analysis  methodology  may  be  most  useful  for  stable  task 
environments  where  task  performance  is  algorithmic  and  therefore,  predictable  enough  that 
relatively  few  cognitive  demands  are  imposed  on  the  performer.  For  example,  one  can  imagine 
the  prespecified  steps  in  Table  1  being  learned  for  purposes  of  routinized  efficiency  because  of  the 
relatively  simple  device  that  is  the  object  of  the  performance.  However,  on  more  complex 
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devices,  prespecification  of  all  procedural  sequences  is  not  feasible.  Such  tasks  are  ill  structured 
and  thus  it  becomes  important  to  specify  the  conceptual  support  knowledge  and  top-level  control 
knowledge  (for  a  set  of  procedural  steps)  so  that  the  learner  acquires  the  capability  to  make 
informed  decisions  when  forced  to  generate  new  solutions  to  novel  problems.  For  these  reasons, 
the  PARI  methodology  is  viewed  as  complementary  to  the  ISD  approach. 

GOMS  (Goals,  Operators,  Methods,  Selection  Rules)  Analysis 


Purpose 

The  GOMS  analytic  approach  was  developed  for  modelling  a  specific  type  of  task 
performance,  namely,  human-computer  interactions  (Card,  Moran,  &  Newell,  1983).  The 
approach  yields  an  executable  computer  model  of  a  task  which  is  constructed  by  (1)  performing  a 
task  analysis  to  specify  the  constraints  imposed  on  human  behavior  by  the  nature  and  features  of 
the  task  environment  and  (2)  applying  certain  empirically  derived  estimates  of  the  user's 
processing  speed,  short-term  memory  capacity  and  duration,  and  other  cognitive  parameters.  The 
resultant  model  accounts  for  differences  in  task  performance  in  terms  of  the  cognitive  demands 
placed  on  the  user  by  variable  features  of  alternative  task  environments.  A  GOMS  analysis  of  a 
task  such  as  text  editing  (in  the  word  processing  domain)  can  be  used,  for  example,  to  assess  the 
influence  of  alternative  interface  designs  or  documentation  formats  on  task  performance. 

Procedure/Result 


The  information-processing  activity  of  the  user  is  modeled  in  a  GOMS  task  analysis  by  four 
components:  Goals,  Operators,  Methods  for  achieving  goals,  and  Selection  rules  for  choosing 
among  alternative  methods.  Goal  statements  decompose  and  define  the  task  as  a  hierarchy  of 
subgoals,  which  correspond  to  component  subtasks.  For  example,  in  the  text  editing  domain, 
goals  might  range  from  a  top-level  goal  of  Edit  manuscript  to  a  lower-level  subgoal  such  as 
Locate  line.  Each  goal  has  associated  with  it  one  or  more  Methods  for  achieving  it.  Methods  are 
made  up  of  a  sequence  of  Operators  which  are  executed  serially  to  satisfy  the  goal-subgoal 


20 


hierarchy.  In  the  context  of  text  editing,  operators  might  include  elementary  cognitive  (or 
perceptual  or  motor)  actions  such  as  Insert-text,  Scroll-to,  Type,  and  Verify-edit.  The  operator 
sequence  or  method  corresponds  to  a  procedure  for  accomplishing  a  subtask. 

For  example,  if  the  goal  is  to  Insert  Text,  the  user  may  not  know  where  to  make  the 
insertion,  and  so  the  appropriate  method  might  begin  with  a  rather  global  action  to  consult  the 
manuscript  to  locate  the  insertion  spot  (Operator  1).  The  next  action  might  be  to  select  the  target 
location  by  using  one  of  several  methods  available,  for  example,  scrolling  or  jumping  (Operator 
2).  Once  the  spot  is  located,  the  user  would  issue  the  Insert  command  to  the  editor  (Operator  3 
of  the  method).  Then  s/he  either  remembers  the  text  to  be  inserted  or  consults  the  manuscript 
(Operator  4).  Finally,  the  user  types  in  the  new  text  (Operator  5)  (Card  et  al.,  1983). 

The  final  component  of  the  GOMS  model  is  the  set  of  Selection  Rules.  These  rules  are  used 
as  a  control  structure  for  deciding  which  among  the  available  methods  to  use  in  meeting  a 
particular  goal.  In  GOMS  notational  terms,  each  Selection  Rule  takes  the  form,  "If  X  is  true  in 
the  current  task  situation,  then  use  method  M."  (Illustrative  Goals,  Operators,  Methods  and 
Selection  rules  from  a  GOMS  model  are  shown  in  Table  2.)  A  GOMS  model  is  constructed  from 
data  collected  as  a  computer  user  performs  a  task  such  as  editing  a  manuscript.  The  data  may  be 
available  from  videotaped  records  of  the  user's  behavior  and  time-stamped  files  of  keystrokes  or 
from  input  from  other  devices  such  as  a  mouse  or  joystick.  The  data  are  then  coded  into  a 
protocol  of  operator  sequences,  which  yield,  among  other  indicators,  detailed  information  on  the 
time  required  to  achieve  the  component  goals  of  the  task. 

Relationship  to  the  PARI  Approach 

Because  the  GOMS  technique  was  developed  for  the  purpose  of  modelling  computer  tasks, 
much  of  the  performance  being  analyzed  is  cognitive  activity  that  is  not  directly  observable  (e.g., 
decisions  involving  method  selection,  or  the  execution  of  operators  such  as  "verify  edit").  It  is 
important  to  note  however,  that  in  representing  computer  usage  knowledge,  a  GOMS  analysis 
resembles  an  ISD  task  analysis  in  the  sense  that  procedural  or  how-to-do-it-  knowledge 
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Table  2 

Goals,  Operators,  Methods  and  Selection  Rules  from  a  GOMS  model  of  a  text  editing  task 
(Card,  Moran,  &  Newell,  1983).  * 


GOALS: 

GOAL:  Edit  manuscript 

GOAL:  Edit  unit  task 

GOAL:  Acquire  unit  task 

GOAL:  Insert  (insertion  point  key,  new  text) 

GOAL:  Delete  (old  text  key) 

GOAL:  Replace  (old  text  key,  new  text) 

GOAL:  Move  (insertion  point  key,  old  text  key) 

GOAL:  Select  target  (manuscript  position,  position  type,  visual  search  target) 
GOAL:  Point  to  target  (manuscript  position,  visual  search  target,  select?) 
GOAL:  Point  there  (screen  position,  text  type,  select?) 

OPERATORS: 

Get  from  manuscript  (desired  information,  attribute) 

Get  from  display  (desired  information,  attribute,  manuscript  position) 

Scroll  to  (line  in  manuscript) 

Point  (screen  position,  text  type,  select?) 

Jump  to  (line  in  manuscript) 

Insert  text 
Delete  text 
Replace  text 
Type  (new  text) 

Execute  (task) 

Verify  Edit 

METHODS: 

One  at  a  time  method 
Acquire  execute  verify  method 
Read  task  in  manuscript  method 
Insert  command  method 
Delete  command  method 
Replace  command  method 
Delete  insert  method 
Zero  in  method 
Rough  point  method 
Character  point  method 
Word  point  method 
Text  segment  point  method 
Insertion  point  method 
Point  without  scrolling  method 
Scroll  and  point  method 
Jump  method 

SELECTION  RULES: 

Rough  loc  rule 
Text  segment  rule 
Character  point  rule 
Word  point  rule 
Insertion  point  rule 
Top  2/3  rule 
Bottom  1/3  rule 
OffScreen  rule 


*  Note  From  The  Psychology  of  Human  Computer  Interaction  (p.  21 0)  by  Card,  Moran,  &  Newell,  1 983, 
Hillsdale,  NJ:  Erlbaum.  Copyright  1983  by  L.  Erlbaum  Associates.  Reprinted  with  permission. 
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dominates  the  model.  In  addition,  certain  types  of  knowledge  are  assumed  to  be  irrelevant  to  the 
human-computer  interaction.  For  example,  system  knowledge  (i.e.,  how  the  computer  works)  is 
not  required  to  interact  effectively  with  the  system.  Nor  is  knowledge  of  the  domain  to  which  the 
computer  task  applies  considered  relevant.  One  needn't  necessarily  have  domain  knowledge 
about  rainfall  in  South  America  in  order  to  edit  a  manuscript  on  the  subject.  The  assumption  is 
that  such  knowledge  does  not  affect  the  ability  of  the  user  to  interact  with  the  system.3  Thus,  a 
GOMS  analysis  models  human-computer  interaction  as  relatively  domain-independent  procedural 
skills.  The  PARI  methodology  can  be  considered  an  extension  of  this  approach  with  its 
applicability  to  knowledge  rich  tasks  where  three  types  of  knowledge  are  coordinated  for  task 
performance  —  the  procedural  skills  themselves  plus  underlying  conceptual  support  knowledge 
consisting  of  system  and  strategic  knowledge  structures. 


The  Knowledge-Engineering  Approach 


Purpose 

Like  a  GOMS  analysis,  the  goal  of  the  knowledge  engineering  approach  is  to  produce  an 
executable  (computational)  model  for  performing  cognitive  tasks  of  interest.  Applications  have 
typically  been  in  the  development  of  expert  systems,  which  are  designed  to  aid  nonexperts  in  task 
performance  and  decision  making.  Examples  of  domains  targeted  in  expert  system  development 
include  medical  diagnosis  (e.g.,  MYCIN  [Shortliffe,  1976];  MDX  [Chandrasekaran  et  al.,  1979]; 
Puff  [Feigenbaum,  1977];  KMS  [Reggia  et  al.,  1980]),  oil  and  mineral  exploration  (e.g.,  Dipmeter 
Advisor  [Davis  et  al.,  1981];  Prospector  [Hart  et  al.,  1978],  computer  configuration  (e.g.,  R1 
[McDermott  and  Steel,  1981],  and  analysis  of  electrical  circuits  (e.g.,  EL  [Stallman  &  Sussman, 
1977]). 


3  However,  see  Kieras  and  Poison  (1985)  for  a  description  of  how  procedural  knowledge  that  is 
captured  in  a  GOMS  analysis  can  be  used  to  define  device  knowledge  that  is  relevant  for  a  task, 
i.e.,  device  knowledge  which  allows  the  user  to  infer  the  exact  operating  procedures  of  the 
system. 
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It  is  important  to  underscore  the  use  of  expert  systems  as  performance  aids,  not  instructional 
systems.  This  intended  utility  influences  the  types  of  knowledge  that  are  elicited  via  knowledge 
engineering  techniques.  More  specifically,  the  role  of  an  expert  system  is  to  execute  a  task,  not 
explain  why  the  task  is  executed  in  a  particular  way  for  purposes  of  human  learning.  As  a 
consequence,  conceptual  support  knowledge  that  provides  reasons  for  task  execution  is  of  little 
interest  to  the  knowledge  engineer. 

Procedures/Results 


The  knowledge  engineering  process  can  be  conceived  in  terms  of  five  stages:  identification. 
in  which  the  subject-matter  expert  and  a  knowledge  engineer  work  together  to  determine  the 
characteristics  of  the  problem(s)  to  be  addressed  by  the  expert  system;  conceptualization,  in  which 
the  key  concepts  in  the  domain  of  interest,  i.e.,  components  of  system  or  declarative  knowledge, 
are  explicated,  along  with  concept  interrelationships  and  the  information-flow  characteristics 
needed  to  describe  problem  solving  in  the  domain;  formalization,  in  which  the  key  concepts  and 
relations  are  transformed  into  a  formal  representation  (computer)  language  (i.e.,  domain-specific 
concepts  are  defined  in  the  language  of  the  expert  system);  implementation,  in  which  the 
procedural  operations  and  reasoning  processes  (e.g.,  rules,  frames,  networks,  or  predicate 
calculus)  that  make  use  of  the  concepts  are  formulated;  and  finally,  testing,  in  which  the  resultant 
executable  expert  model  is  evaluated  and  revised  to  conform  to  the  performance  standards 
established  by  the  human  experts  in  the  problem  domain  (Hayes-Roth,  Waterman,  &  Lenat, 

1983). 

Two  approaches  to  knowledge  engineering  have  dominated  expert  system  development  of 
this  type.  The  traditional  approach,  verbal  protocol  analysis4,  involves  knowledge  elicitation 
grounded  in  observable,  interrogatory,  and  intuitive  techniques  (Waterman,  1985).  Experts  are 
asked  to  think  aloud  as  they  solve  problems  or  to  specify  what  they  know  about  a  given  concept. 


4  The  use  of  verbal  protocol  analysis  has  not  been  restricted  to  expert  system  development  but 
extends  to  the  study  of  problem  solving  in  general.  These  methods  will  therefore  be  discussed  in 
detail  in  a  later  section. 
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A  knowledge  engineer  probes  the  expert  for  explicit  information  regarding  problem  solving 
performance  and  has  the  responsibility  for  representing  the  expert's  task  execution  according  to 
some  predetermined  formalism  or  representational  (computer)  language.  Table  3  illustrates  raw 
data  from  a  verbal  protocol  session  represented  as  condition-action  rules.  In  expert  system 
development  of  this  type,  the  knowledge  engineer  is  usually  not  an  expert  in  the  subject  matter 
domain  but  has  the  computer  skills  required  for  programming  the  knowledge  base  in  the  chosen 
representation  language. 

A  second  approach  to  knowledge  engineering  is  the  use  of  automated  tools  for  knowledge 
acquisition  and  validation  or  testing  of  the  resultant  knowledge  base.  The  purpose  of  automating 
the  knowledge  engineering  task  is  to  allow  the  expert  to  interact  directly  with  a  computer  system 
rather  than  work  through  an  intermediate  layer,  i.e.,  a  human  knowledge  engineer.  Automated 
tools  embody  certain  underlying  assumptions  about  the  particular  representational  scheme  to  be 
used  by  the  expert  system  (e.g.,  rules,  frames,  networks,  predicate  calculus)  and  the  control 
strategy  employed  for  searching  the  knowledge  base.  In  other  words,  the  expert  does  not  have  to 
specify  how  knowledge  is  used  by  the  system  since  it  is  inherent  in  the  representational  structure 
and  control  mechanisms.  This  approach  allows  the  expert  to  focus  exclusively  on  generating  the 
knowledge  required  to  produce  expert  solutions  and  eliminates  the  need  for  a  knowledge  engineer 
with  sophisticated  programming  expertise,  since  the  knowledge  acquisition  tool  automatically 
transforms  the  input  into  the  representational  language  that  is  to  be  used  in  the  targeted  expert 
system.  The  choice  of  a  representational  scheme  in  expert  system  development  involves 
consideration  of  which  scheme  yields  the  most  efficient  system  performance.  That  choice,  in  turn, 
gives  structure  to  both  the  elicitation  and  interpretation  of  domain  knowledge.  To  date,  the  most 
successful  expert  systems  embody  rule-based  representation  schemes  (Boose,  1986). 

Relationship  to  the  PARI  Approach 

We  noted  in  an  earlier  section  that  the  representational  structure  used  in  an  expert  system 
may  not  be  compatible  with  the  goal  of  teaching.  Recall  that  the  limitations  of  GUIDON  as  a 
tutor  had  primarily  to  do  with  the  rule-based  representation  of  domain  knowledge  in  MYCIN,  the 
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Table  3 

Verbal  protocol  data  and  resulting  procedural  rules  (adapted from  Buchanan  et  al.,  1983). 


RAW  VERBAL  PROTOCOL  DATA 

(From  interview  on  identifying  chemical  spills  and  their  sources,  Buchanan  et  al.,  1983) 


KNOWLEDGE  ENGINEER: 

Suppose  you  were  told  that  a  spill  had  been  detected  in  White  Oak  Creek  one  mile  before  it 
enters  White  Oak  Lake.  What  would  you  do  to  contain  the  spill? 

EXPERT: 

That  depends  on  a  number  of  factors.  I  would  need  to  find  the  source  in  order  to  prevent  the 
possibility  of  further  contamination,  probably  by  checking  drains  and  manholes  for  signs  of  the 
spill  material.  And  it  helps  to  know  what  the  spill  material  is. 

KNOWLEDGE  ENGINEER: 

How  can  you  tell  what  it  is? 

EXPERT: 

Sometimes  you  can  tell  what  the  substance  is  by  its  smell.  Sometimes  you  can  tell  by  its  color, 
but  that's  not  always  reliable  since  dyes  are  used  a  lot  nowadays.  Oil,  however,  floats  on  the 
surface  and  forms  a  silvery  film,  while  acids  dissolve  completely  in  the  water.  Once  you 
discover  the  type  of  material  spilled,  you  can  eliminate  any  buildings  that  either  don't  store  the 
material  at  all  or  don't  store  enough  of  it  to  account  for  the  spill. _ 

RAW  DATA  CONVERTED  TO  CONDITION-ACTION  RULES 


To  determine  spill  material: 

[1]  If  the  spill  does  not  dissolve  in  water  and  the  spill  does  form  a  silvery  film,  let  the  spill  be 
oil. 

[2] If  the  spill  does  dissolve  in  water  and  the  spill  does  form  no  film,  let  the  spill  be  acid. 

[3]  If  the  spill  =  oil  and  the  odor  of  the  spill  is  known,  choose  situation: 

•  if  the  spill  does  smell  of  gasoline,  let  the  material  of  the  spill  be  gasoline  with  a 
certainty  .9; 

•  if  the  spill  does  smell  of  diesel  oil,  let  the  material  of  the  spill  be  diesel  oil  with 
certainty  .8. 

[4]  If  the  spill  =  acid  and  the  odor  of  the  spill  is  known,  choose  situation: 

•  if  the  spill  does  have  a  pungent/choking  odor  let  the  material  of  the  spill  be 
hydrochloric  acid  with  certainty  .7; 

•  if  the  spill  does  smell  of  vinegar  let  the  material  of  the  spill  be  acetic  acid  with 

_ certainty  .8. _ 


expert  system  whose  knowledge  the  tutor  was  intended  to  convey  to  students.  MYCIN  did  not 
require  knowledge  of  human  system  functioning,  nor  did  it  require  human-like  reasoning 
strategies  to  perform  successfully.  Without  such  knowledge,  however,  students  were  unable  to 
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acquire  and  retain  MYCIN's  rules.  The  point  here  is  that  the  form  of  data  resulting  from  a  task 
analysis  for  expert  system  development  may  be  of  limited  use  if  one's  goal  is  to  teach  a  human 
how  to  execute  the  task  rather  than  to  program  a  computer  to  execute  it. 

Although  knowledge-engineering  tools  and  the  PARI  methodology  were  developed  for 
different  purposes,  they  do  have  one  critical  feature  in  common;  that  is,  the  type  of  problem¬ 
solving  activity  that  is  typically  the  focus  of  the  knowledge  acquisition  process.  Both  procedures 
examine  problems  that  are  ill  structured:  their  solutions  are  not  based  on  algorithms  but  on  a 
collection  of  informal  knowledge  acquired  through  experience.  The  design  of  certain  knowledge¬ 
engineering  tools  (e.g.,  TEIRESIAS;  Davis,  1982)  and  the  PARI  procedures  have  been  influenced 
in  the  same  way  by  this  common  focus. 

More  specifically,  both  methods  emphasize  the  utility  of  knowledge  acquisition  in  the 
context  of  solving  particular  problems.  Davis  (1982)  notes  that  having  experts  articulate  their 
knowledge  while  solving  actual  problems  is  especially  important  when  the  performance  program 
is  designed  to  accommodate  inexact  knowledge  in  domains  where  knowledge  has  not  been 
extensively  formalized.  In  such  domains,  experts  will  often  be  required  to  formally  codify  pieces 
of  knowledge  for  the  first  time.  Allowing  experts  to  articulate  their  knowledge  as  it  relates  to 
particular  problems  situates  the  knowledge  acquisition  process  and  makes  the  resulting  knowledge 
base  more  robust. 


Verbal  Protocol  Analysis 


Purpose 

Verbal  protocol  analysis  is  a  general  term  used  to  describe  knowledge  elicitation  techniques 
in  which  a  researcher,  task  analyst,  or  knowledge  engineer  interacts  with  an  expert  in  order  to 
document  the  expert's  knowledge  base  and  knowledge  deployment  during  performance.  While 
the  verbal  protocol  approach  is  commonly  used  by  knowledge  engineers  in  the  development  of 
expert  systems,  the  use  of  verbal  data  to  study  human  performance  has  its  roots  in  psychological 
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research  studies  of  problem  solving.  Pioneers  in  developing  the  approach  believed  that  only  the 
richness  of  verbal  data  could  adequately  reflect  the  complexity  of  knowledge  and  the  reasoning 
processes  used  by  humans  in  complex  problem  solving  tasks.  (Newell  &  Simon,  1972;  Ericsson  & 
Simon,  1980). 

The  adaptation  of  verbal  protocol  methods  to  expert  system  development  has  consistently 
moved  in  the  direction  of  prespecifying  a  given  representation  scheme  for  knowledge  elicitation, 
the  dominant  one  being  condition-action  rules;  however,  there  is  more  flexibility  in  verbal 
protocol  methods  than  one  may  expect.  Further,  the  richness  of  the  data  captured  in  verbal 
protocols  makes  the  approach  particularly  useful  for  studying  domains  in  which  expertise  has  not 
been  extensively  formalized  or  is  ill  structured. 

Procedure/Result 

Wielinga  and  Breuker  (1985)  describe  five  basic  methods  for  eliciting  verbal  data.  Since  the 
methods  complement  each  other  with  respect  to  the  type  of  information  they  yield,  a  combination 
of  these  methods  is  typically  used  in  any  given  application.  In  a  focused  interview,  the  expert 
answers  questions  from  an  agenda  established  by  the  analyst  whose  goal  is  to  acquire  an  overview 
of  the  domain  or  task.  The  interview  focuses  on  the  domain,  the  functions  of  expertise,  the  job 
environment,  and  characteristics  of  the  user  of  the  prospective  expert  system.  In  focusing  on 
general  issues,  the  interview  provides  the  basis  for  later,  more  detailed  discussions  in  a  structured 
interview.  In  a  structured  interview,  the  researcher  probes  the  expert  for  detailed  explanations  of 
general  concepts.  Here,  the  purpose  is  to  provide  the  researcher  with  deeper  insight  into  the 
structure  of  domain  concepts  and  their  interrelationships  through  queries  about  the  static  aspects 
of  the  domain,  including  never-changing,  indisputable  facts,  theories,  and  conceptual  objects. 

By  contrast,  the  dynamic  aspects  of  the  domain  are  encountered  through  actual  task 
performance,  for  example,  the  dynamic  reasoning  processes  engaged  in  by  experts  during  solution 
searches.  These  processes  are  more  easily  captured  in  introspective  reports  where  the  expert 
describes  how  hypothetical  problems  would  be  solved,  and  in  self-reports  in  which  the  expert 
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thinks  aloud  as  s/he  actually  solves  problems  under  simulated  conditions.  Problem  solving  may 
also  be  observed  during  user  dialogues  where  an  expert  answers  the  questions  of  a  prospective 
user  of  the  proposed  knowledge  base.  Finally,  the  expert  may  be  asked  to  review  protocols 
obtained  earlier  to  provide  necessary  elaborations  and  to  fill  in  gaps  in  the  data. 

The  richness  of  verbal  protocol  data  arises  from  the  relatively  few  constraints  placed  on  the 
questions  from  the  analyst  or  on  the  responses  of  the  expert.  This  procedural  looseness  as  well  as 
other  features  of  verbal  protocol  methods  have  not  escaped  criticism,  however.  The  introspective 
nature  of  the  data  captured  in  verbal  protocols  has  been  criticized  by  researchers  on  several 
grounds  (Nisbett  &  Wilson,  1977).  First,  because  subjects  cannot  be  assumed  to  have  conscious 
access  to  the  intermediate  stages  of  processing,  the  verbal  data  they  report  during  problem  solving 
may  not  correspond  to  the  actual  internal  representations  they  use.  Secondly,  verbal  reporting 
during  problem  solving  may  artificially  influence  the  form  and  content  of  the  data  by  requiring 
subjects  to  verbalize  events  that  would  not  normally  be  reported  during  task  performance. 

Finally,  because  of  the  flexibility  of  language,  different  subjects  may  express  the  same  thoughts  in 
idiosyncratic  ways,  making  the  interpretation  of  verbal  data  difficult.  Thus,  introspective  data 
have  been  seen  by  critics  as  useful  only  for  generating  hypotheses  concerning  the  nature  of 
psychological  processes,  but  not  for  their  verification. 

Relationship  to  the  PARI  Approach 

The  PARI  methodology  produces  the  rich  sort  of  verbal  data  needed  to  capture  relevant 
knowledge  and  shed  light  on  the  processes  used  in  certain  types  of  problem  solving.  However,  it 
incorporates  several  features  designed  to  obviate  the  criticisms  noted  above.  First,  the  data  are 
collected  m  situ  as  subjects  think  aloud  while  they  solve  problems.  Subjects  are  not  asked  to 
respond  retrospectively  to  questions  which  can  encourage  them  to  speculate  and  draw  inferences 
about  their  thought  processes.  Concurrent  verbalization  produces  more  detail  in  subjects' 
descriptions  of  their  thoughts  since  what  is  remembered  decreases  with  the  delay  in  recall. 
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The  PARI  procedure  also  provides  a  structure  for  the  interview  which  the  researcher  or 
analyst  uses  as  a  framework  for  probing  subjects:  for  each  step  in  the  problem  solution,  the  solver 
is  asked  to  identify  the  action  being  taken  at  that  step,  the  goal  or  cognitive  precursor  to  the 
action,  and  after  being  told  the  result  of  the  action,  is  asked  to  interpret  the  result.  This  structure 
was  derived  empirically  from  observation  of  dozens  of  technicians  engaged  in  electronic  fault 
isolation  tasks  (Means  &  Gott,  1988;  Gott,  1987).  These  technicians'  natural  approach  to 
troubleshooting  is  to  take  some  action  or  series  of  actions  based  on  an  hypothesis  about  the  fault 
location  which  is  derived  from  some  internalized  mental  representation  of  how  the  circuit  works. 
Experts  also  deploy  a  strategy  to  investigate  the  circuit  that  will  allow  them  to  narrow  down  the 
problem  space  and  then  to  interpret  the  outcomes  of  their  actions  in  terms  of  current  hypotheses 
and  the  appropriate  next  steps.  Their  use  of  mental  models  to  construct  solutions  appears  to  be 
something  that  experts  can  readily  talk  about  and  represent  in  diagrams.  Whether  or  not  the  data 
reflect  all  of  the  features  of  subjects'  internal  representations  is  not  clear.  However,  it  is  our  belief 
that  the  data  captured  in  these  studies  yield  multiple  levels  of  description  that  are  most  useful  for 
identifying  instructional  targets  for  teaching  adaptive  problem  solving. 

A  second  advantage  of  using  a  structured  interview  technique  like  the  PARI  procedure  is 
that  the  data  are  collected  systematically.  Such  a  framework  is  particularly  useful  in  large-scale 
research  where  many  individuals  may  be  collecting  the  data.  The  data  are  less  likely  to  be 
influenced  by  inconsistencies  in  the  task  analyst(s)'  approach  to  interviewing  which  can  compound 
the  complexity  of  the  interpretation  process.  The  structure  of  the  interview  also  facilitates 
comparisons  across  subjects,  problems,  and  domains,  since  the  same  basic  questions  are  asked 
under  all  conditions. 


Summary 

The  PARI  methodology  borrows  several  features  from  the  approaches  to  task  analysis 
described  in  this  section.  Like  the  ISD  approach,  our  method  attempts  to  provide  a  practical, 
analytic  tool  that  can  be  used  by  nonscientists  for  instructional  development.  It  focuses,  however, 
on  the  contribution  of  multiple  types  of  tacit  knowledge  to  task  performance  and  the  cognitive 
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processes  that  deploy  the  knowledge.  Like  the  GOMS  approach,  the  PARI  procedure  also 
attempts  to  capture  the  structure  of  a  task  and  the  reasons  underlying  a  particular  sequence  of 
problem-solving  steps.  The  strategic  processes  responsible  for  organizing  the  solution  and  for 
searching  the  knowledge  base  are  critical  components  of  the  performance  models  being  generated 
from  cognitive  task  analyses.  Providing  executable  performance  models  is  the  goal  of  a  GOMS 
analysis  and  of  knowledge  engineering  as  well.  The  PARI  methodology  is  consistent  with  this 
approach  and  acknowledges  the  importance  of  modelling  performance  at  varying  proficiency 
levels  for  the  purpose  of  identifying  instructional  targets  and  learning  trajectories.  In  order  to 
capture  the  richness  of  technical  knowledge  and  the  full  complexity  of  problem  solving  by  human 
experts,  a  verbal  protocol  method  was  adopted  for  the  PARI  procedures.  The  following  section 
describes  the  procedure  in  greater  detail. 
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m.  THE  PARI  METHODOLOGY:  DATA  COLLECTION 


In  this  section,  we  describe  how  PARI  data  are  collected  and  discuss  the  relationship 
between  this  technique  and  the  theoretical  framework  presented  in  Section  I  of  this  guide.  The 
PARI  methodology  is  a  procedure  for  examining  cognitive  tasks  that  involve  the  interaction  of  a 
human  problem  solver  with  a  complex  system.5  The  procedure  revolves  around  a  structured 
interview  during  which  problem  solvers  in  the  domain  of  interest  are  asked  to  think  aloud  while 
they  solve  authentic  problems.  The  interview  is  structured  to  simulate  the  actual  problem  solving- 
task  environment  and  to  elicit  the  performer's  knowledge  of  the  system,  problem  solving 
procedures  and  operations,  and  strategic  control  processes,  including  planning,  knowledge 
deployment,  and  performance  regulation/monitoring  (see  Figure  1).  The  intent  of  this  problem- 
situated  approach  is  to  reveal  knowledge  and  skill  in  the  context  of  their  use;  therefore,  problem 
solvers  are  probed  extensively  to  make  explicit  the  reasons  for  the  content  and  organization  of 
their  solutions.  Once  the  reasons  are  made  explicit,  they  become  knowable  by  learners  and 
concrete  targets  for  training. 


Overview  of  the  PARI  Procedures 

Stages  of  the  PARI  Methodology 

The  PARI  data  collection  procedures  comprise  nine  stages,  the  first  four  of  which  are 
preparatory  to  the  basic  PARI  structured  interviews  (see  Table  4  ).  In  general,  the  first  stages 
establish  the  sample  of  experts  that  will  contribute  to  the  task  analysis  and  identify  at  a  general 
level  the  problem  solving  tasks  and  associated  cognitive  skills  that  are  to  be  considered  as 
instructional  targets.  These  tasks  and  skills  serve  as  the  initial  foci  in  the  development  of  specific 
problems  to  be  solved  during  the  PARI  interviews.  The  final  five  stages  of  the  procedure  involve 


5  Although  the  PARI  methodology  was  developed  in  the  context  of  studying  electronic 
troubleshooting  (which  involves  interacting  with  a  technologically  complex  system),  we  believe 
the  procedure  can  be  generalized  to  the  study  of  other  types  of  human-system  interactions. 


32 


the  basic  expert  and  novice  interview  sessions,  the  followup  rehashes,  and  reviews  of  the  data  by 
experts. 

Table  4 

Stages  of  PARI  data  collection  procedures. 


Stage  I:  Identification  of  experts  and  orientation  of  researchers 

Stage  II:  Focus  of  training  established 

Stage  III:  Generation  and  consolidation  of  problem  types 

Stage  IV:  Problem  category  assignment  and  problem  design 

Stage  V:  Anticipation  of  PARI  solution  paths 

Stage  VI:  Generation  of  expert  solutions 

Stage  VII:  Problem  set  review  by  expert  problem  solvers 

Stage  VIII:  Generation  of  problem  solutions  by  intermediate  and  novice  technicians 
Stage  IX:  Problem  set  review  by  independent  (advanced)  experts 


The  PARI  Structure 


The  cornerstone  of  the  methodology  is  an  expert  problem-solving  dyad.  One  expert  poses  a 
problem  and  simulates  equipment  responses  to  a  second  expert,  who  attempts  to  (verbally)  isolate 
the  fault  which  has  been  conceived  by  the  first  expert.  The  dyad  format  is  then  extended  by 
pairing  intermediate  and  novice  technicians  with  an  expert  who  poses  a  problem  and  simulates 
equipment  responses  for  each  problem  solver.  During  each  interview,  the  role  of  the 
researcher/task  analyst  is  to  record  the  problem  solver's  solution  steps  as  discrete  operations  or 
Actions,  e.g.,  "trace  schematic  for  XYZ  circuit  card"  or  "measure  voltage  at  pin  28."  Then  the 
solver  is  probed  by  the  analyst  to  express  the  reasons  (or  Precursors!  for  the  actions.  A  Precursor 
reveals  the  particular  goal  or  intent  of  the  solver  in  executing  each  action;  in  turn,  a  sequence  of 
precursors  reveals  the  performer's  top  level  plan  or  goal  structure.  This  is  the  glue  that  bonds  the 
detailed  steps  together.  The  analyst  also  probes  the  performer  for  an  Interpretation  of  the  system 
response  that  is  provided  by  the  expert  who  is  posing  the  problem.  Finally,  the  analyst  asks  the 
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problem  solver  to  draw  a  block  diagram  level-sketch  of  the  relevant  equipment  to  illustrate  each 
solution  step.  The  resultant  series  of  sketches  reveals  the  evolving  mental  models  of  the 
equipment  used  by  the  performer  to  guide  his/her  movement  through  the  problem  space. 

Sequences  of  mental  events  such  as  these  are  called  PARI  structures  (Precursor  [to  Action] 
—  Action  —  Result  --  Interpretation).  An  example  node  from  a  PARI  trace  (i.e.,  a  single  solution 
step)  is  shown  in  Figure  3.  After  a  basic  trace  is  recorded,  a  series  of  "rehashes"  is  conducted  by 
the  researcher/analyst  to  verify  and  elaborate  the  solution  trace.  To  complete  the  process,  an 
independent  set  of  experts  reviews  the  problem  set  to  judge  its  completeness  and 
representativeness,  and  the  expert  participants  themselves  rate  the  criticality  of  the  cognitive  skills 
required  to  solve  each  problem.  We  turn  now  from  this  overview  to  examine  each  of  the  data 
collection  stages  in  more  detail.  Our  particular  focus  will  be  on  the  theoretical  and  empirical 
rationale  for  each  stage. 

PRECURSOR 

I  want  to  see  if  the  LRU  ID  resistor  is  good. 

ACTION 

Remove  the  cable  from  J12  of  the  LRU  and  ohm  out  the 

path  through  the  LRU  from  pin  68  to  pin  128. 

RESULT 

The  reading  is  1.55  Mohms. 


INTERPRETATION 

The  problem  isn't  in  the  LRU,  it's  in  the  test  station 
or  the  test  package. 
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Stage  I:  Identification  of  Experts  and  Orientation  of  Researchers 

Goal  and  Rationale 


The  identification  of  experts  to  participate  in  the  PARI  sessions  involves  several 
considerations  that  are  central  to  the  goals  of  the  task  analysis.  First,  the  development  of  a 
cognitive  model  of  skilled  performance  in  a  domain  of  ill-structured  tasks  requires  input  from 
multiple  experts.  The  very  nature  of  adaptive  expertise  entails  the  ability  to  consider  a  wide  range 
of  alternative  solution  strategies  and  associated  procedures  for  implementing  those  strategies, 
based  on  one's  knowledge  of  the  system.  Because  different  experts  have  different  types  and  levels 
of  system  knowledge,  as  well  as  varying  propensities  to  deploy  different  types  of  procedures, 
problem  solving  will  vary  from  expert  to  expert.  Any  one  expert  may  exhibit  neither  the  full  range 
of  strategic  and  procedural  options  nor  all  of  the  device  models  that  would  inform  a  robust 
instructional  system  for  complex  problem  solving. 

In  addition  to  providing  multiple  perspectives  on  problem  solving,  the  expert  participants 
must  be  sensitive  to  the  training  needs  of  novices  in  the  field.  Input  from  experts  regarding  the 
learning  impediments  of  novices  helps  to  determine  the  curriculum  issues  to  be  addressed  by  the 
training.  Although  some  experts  will  have  useful  insights  into  training  needs  based  on  their  own 
earlier  experiences  as  trainees,  it  is  beneficial  if  the  selected  individuals  have  had  recent,  direct 
responsibilities  for  training  less-experienced  personnel  in  an  on-the-job  training  context.  This 
experience  can  contribute  to  the  validity  of  the  training  foci,  which  are  established  in  Stage  II,  and 
to  the  problems  that  are  developed  in  Stages  m  and  IV  as  the  foundation  of  the  training. 

The  orientation  phase  of  Stage  I  is  intended  to  serve  two  purposes.  First,  it  is  designed  to 
expand  the  researchers'  domain  knowledge  so  that  the  (fault  isolation)  problems  that  will  be 
generated  and  solved  in  the  later  stages  of  the  workshop  can  be  reasonably  well  understood. 
Secondly,  it  provides  an  additional  opportunity  for  the  capabilities  of  the  identified  experts  to  be 
evaluated. 
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Procedure 


Procedurally,  the  first  step  in  identifying  expert  participants  (in  our  project)  has  been  to  have 
supervisors  at  a  given  work  site  select  their  best  hands-on  technicians  for  participation.  The 
research  team  then  interviews  each  technician  to  determine  his/her  ability  to  present  technical 
information  in  a  coherent  manner,  to  convey  technical  information  to  a  nontechnical  audience,  and 
to  participate  productively  in  the  research  effort.  More  specifically,  each  expert  is  asked  to 
provide  information  about  their  personal  training  and  work  experience  (see  Appendix  A  for  an 
illustrative  example  of  a  training  and  experience  questionnaire),  and  is  then  interviewed  about  the 
equipment  systems  that  are  maintained  in  the  job  in  question. 

The  orientation  of  researchers  begins  by  having  each  expert  generate  a  description  of  the 
equipment  system  that  is  the  primary  focus  of  her/his  job.  Experts  are  instructed  to  begin  at  a 
fairly  general  level  of  description  and  to  draw  illustrative  block  diagrams  that  show  the 
components  being  described  and  their  interrelationships.  Increasingly  detailed  descriptions  of  the 
system  are  then  elicited  by  having  the  expert  iteratively  analyze  each  component  illustrated  in  the 
preceding  description  and  draw  any  new  components  mentioned.6  Once  the  expert  has  described 
the  system  in  as  much  detail  as  possible,  s/he  is  asked  to  talk  about  major  physical  and  functional 
components  of  the  system  and  their  interactions.  Finally,  each  expert  is  asked  to  discuss  examples 
of  typical  problems  (malfunctions)  encountered  with  this  system.  This  step  is  intended  to  prime 
the  experts  for  discussions  about  representative  equipment  problems  that  would  provide  a  solid 
basis  for  training. 

The  equipment  orientation  is  concluded  with  a  meeting  at  which  the  experts  orient  the 
researchers  to  the  site-specific  workplace  ecology,  that  is,  conditions  and  practices  of  the 


6  An  automobile  analogy  is  used  to  illustrate  how  one  might  generate  a  description  of  a  system  at 
increasingly  detailed  levels  of  analysis:  at  the  major  functional  component  level,  an  automobile 
can  be  described  as  drive  train,  body,  and  frame.  At  a  lower  level,  the  drive  train  can  be  described 
as  engine,  transmission,  axle  assemblies,  etc.  At  an  even  lower  level,  the  engine  can  be  described 
as  pistons,  cylinders,  etc.  One  can  proceed  in  this  manner  until  the  nuts  and  bolts  level  is  reached, 
if  desired. 
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workplace  or  requirements  of  the  job  that  influence  how  the  job  is  performed.  A  questionnaire 
relevant  to  the  workplace  ecology  of  the  aircraft  maintenance  jobs  studied  under  the  BJS  project 
is  provided  in  Appendix  B. 

Result  of  Stage  I 

Stage  I  is  concluded  when  the  research  team  makes  a  final  selection  of  the  experts  to 
participate  in  the  PARI  workshop.  It  has  been  our  experience  that  the  quality  of  expertise  needed 
to  conduct  an  effective  cognitive  task  analysis  is  quite  rare.  For  example,  in  a  technical  work 
cadre  of  25  to  30  individuals,  we  have  found  that  generally  only  one  or  two  workers  meet  our 
criteria.  These  bona  fide  experts  have  on  average  8  to  10  years  experience  in  the  domain  and  are 
still  actively  engaged  in  hands-on  problem  solving,  that  is,  they  have  not  moved  into 
management/administrative  positions.  We  have  found  that  the  best  PARI  products  result  when 
the  highest  levei  expertise  at  a  site  is  involved  in  the  workshop,  even  when  that  means  that  only 
two  experts  per  site  (the  minimum)  participate  and  that  the  research  team  will  have  to  travel  to  a 
second  and  even  a  third  site  to  obtain  input  from  a  sufficient  number  of  individuals  (usually  six  to 
eight). 


Stage  II:  Focus  of  Training  Established 


Goal  and  Rational 


One  of  the  basic  assumptions  that  underlies  the  PARI  methodology  is  particularly  salient  for 
Stage  II,  namely,  that  results  of  the  task  analysis  will  be  used  to  develop  training  that  targets 
complex  problem  solving.  The  purpose  of  Stage  II  is  to  have  the  experts  selected  in  Stage  I 
identify  the  cognitively  complex  tasks  within  a  job  and  specify  the  general  nature  of  the  associated 
cognitive  demands.  These  instructional  foci  influence  all  subsequent  task  analysis  activities. 
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Procedure 


In  the  study  of  Air  Force  job  specialties,  a  good  first  step  is  to  have  experts  examine  the 
results  of  occupational  surveys  and  Specialty  Training  Standards.  These  reports  provide  an 
inventory  of  the  general  duties  and  the  more  specific  job  tasks  required  by  a  technical  specialty. 
They  also  provide  information  concerning  the  frequency  with  which  tasks  are  performed  as  well 
as  their  level  of  difficulty  (defined  in  terms  of  how  long  it  takes,  on  the  average,  for  a  person  to 
learn  the  task).  For  example,  for  aircraft  maintenance  jobs,  the  occupational  surveys  might  list 
duties  such  as  maintaining  radar  systems,  maintaining  optical  sight  or  integrated  display  systems, 
maintaining  low  altitude  radar  altimeter  (LARA),  and  so  forth.  Within  those  duty  categories  (the 
last  one  for  example),  tasks  listed  might  include,  "isolate  malfunctions  to  LARA  calibrator  units," 
or  "perform  operational  checks  of  LARA  systems." 

The  task  inventory  and  the  associated  frequency  and  difficulty  ratings  are  used  by  the 
experts  and  research  team  to  guide  the  identification  of  maintenance  tasks  that  warrant  a  cognitive 
analysis,  i.e.,  tasks  that  have  sufficient  cognitive  complexity.  Two  related  criteria  are  used  in 
judging  the  cognitive  complexity  of  tasks:  the  stability  of  the  task  (or  system)  environment  in 
which  problem  solving  occurs  and  the  number  of  decisions  required  in  performing  the  task.  An 
unstable  job  environment  is  characterized  by  unpredictable  events  or  conditions  to  which  the 
problem  solver  must  respond.  By  definition,  an  extremely  complex  system  presents  an  unstable 
task  environment  because  of  the  large  number  of  possible  malfunctions  that  may  occur.  Because 
it  is  difficult  to  anticipate  all  possible  equipment  conditions,  it  is  correspondingly  difficult  to 
prespecify  all  possible  sequences  of  solution  steps.  As  a  consequence,  there  are  often  no  well- 
established  procedures  for  task  performance,  and  thus  technicians  are  confronted  with  ill- 
structured  problems.  Even  when  established  procedures  exist,  they  might  be  inefficient  or 
inadequate  under  slightly  altered  conditions. 

The  number  and  complexity  of  required  decisions  are  in  turn  closely  tied  to  the  ill-structured 
nature  of  the  task.  Decisions  may  be  numerous  and/or  complex  because  of  the  number  of 
alternative  choices  available,  because  of  the  number  of  factors  that  must  be  considered  in  making 
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the  decision,  or  because  the  relative  importance  of  various  factors  changes  with  task  conditions. 
For  example,  in  troubleshooting  aircraft  systems,  skilled  technicians  continuously  consider  the 
ratio  of  cost  to  benefit  in  deciding  when  and  how  to  pursue  a  particular  subgoal.  The  actions  they 
elect  to  take  are  supported  by  a  cost-benefit  rationale  wherein  time,  effort,  and  risk  to  the 
equipment  and  to  themselves  are  minimized  and  information  value  and  progress  toward  restored 
equipment  functioning  are  maximized. 

Result  of  Stage  II 

Stage  II  is  completed  when  a  consensus  has  been  reached  on  the  cognitively  complex  tasks 
in  the  domain  and  a  general  characterization  of  the  sources  of  the  task-related  learning  and 
performance  difficulties  has  been  generated.  Together,  these  tasks  and  associated  cognitive 
demands  provide  the  initial  training  foci  for  conducting  Stages  in  and  IV. 

Stage  HI:  Generation  and  Consolidation  of  Problem  (Fault)  Types 

Goal  and  Rationale 


With  the  cognitively  complex  tasks  and  associated  leaming/performance  difficulties  from 
Stage  II  providing  the  focus,  experts  are  directed  in  Stage  m  to  generate  an  exhaustive  list  of  the 
equipment  malfunctions  (and  their  causes)  that  can  initiate  task  performance.  The  goal  is  to  have 
experts  independently  generate  and  then  collectively  consolidate  the  instances  of  system  causes 
and  effects  that  are  related  to  the  tasks  from  Stage  II.  Experts  then  group  related  instances  into 
meaningful  fault  (or  problem)  categories.  This  process  can  be  illustrated  with  a  car  analogy. 

A  cognitively  demanding  task  in  automobile  repair  may  be  troubleshooting  a  fail  in  the  car's 
electrical  system.  Presenting  symptoms  may  be  failure  of  the  headlights  to  come  on.  The 
particular  source(s)  of  equipment  malfunction  that  would  trigger  that  troubleshooting  task  might 
include  a  short  in  the  headlight  wiring,  a  bad  fuse,  a  faulty  dimmer  switch,  a  faulty  on-off  switch, 
and  so  forth.  These  various  causes  of  the  manifested  symptoms  are  related  in  a  variety  of  ways. 
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one  of  which  is  the  underlying  device  model  (of  the  automobile  subsystem)  that  a  problem  solver 
would  invoke  while  reasoning  about  the  fail.  In  the  present  example,  a  model  of  an  electrical 
circuit  that  includes  the  concepts  of  wiring,  circuit  components,  and  switches  would  be  required. 
Procedure 


Stage  m  is  conducted  by  having  lists  of  fault  instances  generated  independently  by  each 
expert  and  then  combined  and  consolidated  during  dyadic  and  group  discussion.  Experts  are 
instructed  to  specify  fault  instances  in  cause  and  effect  language  that  will  communicate  a  class  of 
malfunctions  as  opposed  to  a  very  equipment-specific  malfunction.  For  example,  "bad  stimulus 
routing  caused  by  a  stuck  relay"  would  be  preferred  over  "a  stuck  relay  on  the  A8  card." 

The  consolidation  of  malfunction  instances  into  defensible  categories  involves  several  steps. 
First,  experts  are  asked  to  work  in  pairs,  compare  their  independently  generated  lists,  eliminate 
redundancies,  and  agree  to  a  refined,  consolidated  list.  Secondly,  expert  dyads  are  asked  to  group 
related  faults  on  the  refined  list  into  meaningful  categories.  An  organizing  principle  for  the 
categorization  is  proposed:  group  instances  together  if  they  demand  similar  knowledge  and  skills 
for  solution. 

Several  criteria  are  used  to  evaluate  the  resultant  typology,  which  establishes  the  categories 
of  problems  to  be  generated  in  the  following  stages  and  used  as  the  basis  of  training.  First,  a 
problem  category  should  have  face  validity  in  the  sense  that  experts  would  agree  that  this  type  of 
problem  occurs  with  enough  frequency  that  it  is  a  worthy  topic  of  training.  Secondly,  a  problem 
category  should  have  instructional  value  by  virtue  of  exercising  the  cognitive  skills,  including 
equipment  system  knowledge,  established  as  training  foci  in  Stage  H  In  addition,  each  category 
should  be  illustrated  by  one  or  more  example  problems  (causes  and  effects)  to  clearly 
communicate  the  nature  of  the  category. 

From  a  cognitive  theory  perspective,  the  problem  typology  that  results  from  a  consolidation 
and  grouping  of  fault  instances  should  reflect  the  disparate  knowledge  structures  required  for 
expertise  in  the  job.  In  other  words,  a  cognitive  skills  architecture  (such  as  that  shown  in  Figure 
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1)  should  be  imposed  on  the  domain  of  possible  problems  so  that  the  typology  covers  a  range  of 
ho w-it- works,  how-to-do-it,  and  how-to-decide-what-to-do-and-when  knowledge.  To  illustrate, 
in  the  domain  of  airborne  electronics  (avionics),  diagnosing  problems  involving  radio  versus 
lower-frequency  signals  requires  knowledge  specific  to  each  type  of  signal  since  the  signal 
characteristics  have  consequences  for  the  types  of  cables  used  to  carry  the  signals,  the  types  of 
devices  used  to  measure  the  signals,  and  so  forth.  A  cognitive  model  of  lower  frequency  signals 
would  thus  fail  to  include  system  knowledge  and  associated  procedures  that  would  have  to  be 
used  to  investigate  a  problem  involving  radio  frequency  signals.7  Thus,  problems  involving  both 
types  of  signals  must  be  generated  to  represent  the  different  equipment  systems  that  employ  each 
signal  type. 

Result  of  Stage  HI 

Stage  m  is  completed  when  the  refined,  categorized,  and  exemplified  fault  lists  from  all 
pairs  of  experts  are  combined  to  yield  the  final  problem  typology.  Appendix  C  shows  an 
illustrative  problem  typology  generated  by  a  group  of  experts  in  an  avionics  job  studied  under  the 
BJS  program.  The  typology  is  used  in  the  next  stage  to  ensure  that  representative  examples  of  all 
types  of  problems  encountered  in  the  real  world  will  be  generated  as  the  instructional  base. 

Stage  IV:  Problem  Category  Assignment  and  Specific  Problem  Design 
Goal  and  Rationale 


The  problem  typology  (and  exemplar  problems)  from  Stage  HI  is  used  to  guide  this  stage, 
where  the  goal  is  to  have  experts  design,  in  detail,  representative  problems  that  cover  the 


7  Notice  that  the  strategy  we  use  in  Stage  HI  to  elicit  representative  causes  of  problems  turns  on 
the  importance  we  ascribe  to  system  knowledge  in  ill-structured  problem  solving.  More 
specifically,  our  goal  is  to  have  experts  specify  instances  of  system  malfunctioning,  not  instances 
of  troubleshooting  procedures.  This  strategy  reflects  our  underlying  theoretical  position  that 
technical  expertise  evolves  from  robust  device  models  that  can  in  turn  generate  needed  procedural 
and  strategic  knowledge. 
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categories  represented  in  the  typology.  These  problems  are  then  used  during  the  problem-solving 
interviews  in  Stages  V  and  VI. 

Procedure 


Experts  are  matched  or  assigned  to  problem  categories  (from  Stage  HI)  according  to  then- 
level  of  knowledge  about  the  problem  types  or  subsystems  that  the  problems  target.  The 
illustrative  (exemplar)  problems  generated  in  the  previous  stage  may  be  further  developed  or 
different  problems  may  be  conceived  to  represent  the  categories  in  the  typology.  As  part  of 
designing  problems,  experts  are  instructed  to  (a)  document  conditions  and  consequences  of  the 
cause  of  the  problem  in  a  problem  description,  (b)  generate  problem  statements  that  establish 
initial  conditions  and  symptoms  to  present  to  other  individuals  who  will  be  solving  the  problems, 
and  (c)  anticipate  the  supporting  technical  documentation  (e.g.,  test  procedures,  schematics)  that 
will  be  required  by  others  during  the  solution  process. 

Experts  are  provided  the  following  general  guidelines  for  developing  a  problem  for  each  of 
their  assigned  (or  selected)  categories:  the  problem  should  represent  the  problem  category  well 
by  directly  exercising  critical  skills  and  knowledge  required  for  task  and  job  performance;  the 
problem  should  be  intellectually  (as  opposed  to  physically)  challenging,  and  thereby  have 
instructional  value  as  a  learning  activity;  and  the  problem  should  be  a  good  test  of  problem  solving 
proficiency,  that  is,  not  solvable  by  some  quick  fix  that  averts  major  cognitive  demands. 

After  a  specific  cause  and  effect  are  decided  upon  by  the  expert,  s/he  generates  an  overview 
or  description  of  the  problem.  The  problem  description  specifies  the  job,  task,  and  equipment 
context  of  the  problem;  the  category  that  the  problem  represents  (from  Stage  IE);  the  location  of 
the  fault  in  the  system,  i.e.,  the  cause;  the  symptoms  produced  by  the  fault  (effect);  a  diagram 
illustrating  each  of  the  above;  and  all  anticipated  technical  documentation  required  to  solve  the 
problem.  The  problem  description  also  indicates  how  the  problem  could  be  made  easier  or  more 
difficult  and  describes  the  skills  (e.g.,  signal  tracing  using  schematics,  taking  ohm  measurements, 
etc.)  that  are  likely  to  be  required  in  solving  the  problem. 
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Finally,  a  problem  statement  is  written  to  describe  the  system  conditions  under  which  the 
problem  would  manifest  itself  and  the  symptoms  that  would  be  present  in  the  real  world  were 
such  a  fail  encountered.  This  problem  statement  is  presented  to  individuals  at  the  outset  of  the 
PARI  interview  during  Stage  VI.  It  is  therefore  a  critical  piece  of  the  problem  design  because  it 
initially  establishes  the  authenticity  of  the  problem  context.  The  problem  statement  is 
accompanied  by  any  other  data  related  to  the  fail  that  the  technician  would  have  available  in  the 
real  world,  for  example,  the  status  of  display  panels,  and  which  lights  are  illuminated,  etc. 

Result  of  Stage  IV 

Stage  IV  is  completed  when  the  experts  have  generated  a  problem  description  and  a 
problem  statement  for  each  problem  they  have  designed.  Tables  5  and  6  provide  a  sample 
problem  description  and  the  corresponding  problem  statement,  each  containing  the  elements 
specified  by  the  above  procedure.  The  researcher  should  caution  experts  against  disclosing  to 
other  research  participants  any  information  relating  to  the  problems  they  have  designed.  This  is 
necessary  to  ensure  that  experts  will  be  naive  with  respect  to  the  problems  they  will  solve  in  Stage 
VL 


Stage  V:  Anticipation  of  PARI  Solution  Paths 


Goal  and  Rationale 


After  designing  a  problem  (Stage  IV)  but  prior  to  presenting  the  problem  to  other  experts  to 
solve  (Stage  VI),  the  expert  (problem  designer)  generates  her/his  own  solution  to  the  problem  in  a 
preparatory  PARI  interview.  This  interview  is  preparatory  in  the  sense  that  the  goal  is  to  have  the 
expert  designer  systematically  work  through  possible  solutions  to  her/his  problem  so  that  various 
solution  paths  can  be  anticipated  prior  to  posing  the  problem  to  others.  By  anticipating 
alternative  solutions,  the  designing  expert  is  forced  to  think  through  what  system  responses  (that 
is.  Results')  s/he  will  be  required  to  give  during  PARI  interviews  with  other  performers.  The 
preparatory  interview  also  allows  the  researcher/task  analyst  to  become  familiar  with  the 
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Table  5 

Sample  problem  description. 


JOB:  Electronic  Warfare  Test  Station  Specialist 
PROBLEM  CATEGORY:  Connectors 
CATEGORY  EXEMPLAR:  Pushed  pin 

PROBLEM  DESCRIPTION: 

The  LRU-3  (Low  Band  Receiver  Processor)  is  being  tested.  The  ID  resistor  test  (Test 
segment  1 1,  measurement  1,  test  1)  fails.  The  task  is  to  isolate  the  cause  of  the  failure.  The 
fault  is  a  pushed  pin  at  the  back  of  the  Interface  Chassis  drawer  in  connector  J5.  This 
pushed  pin  is  on  the  path  for  making  ohms  checks  from  the  DMM  out  toward  LRU 
hookups.  The  fault  diagnosis  can  be  made  easier  or  harder  by  moving  the  fault  towards  the 
LRU,  or  back  into  the  test  station  toward  the  DMM,  or  possibly  by  making  the  pushed  pin 
apparent  by  visual  inspection  or  jiggling  the  connections. 

Anticipated  skill  requirements: 

•  Visual  inspections 

•  Swapping 

•  Reseating 

•  Jumpering 

•  Ohms  measurements 

The  tech  data  needed  to  solve  this  problem  include: 

•  TO  12P3-2ALR56-78-1  (LRU  flowchart,  FIG.  5-10) 

•  TO  33D7-50-1-151  (Test  package  schematics,  FIGS.  8-33,  8-23) 

•  TO  33D7-3 8-77-2-2  (Plugboard  map,  FIG.  4-15;  CMG  External  connections,  FIG. 4-40; 
MMX  functional  organization,  FIG.  4-20) 

•  TO  33D7-38-77-2-3  (A/D  console  coaxial  cable  assemblies,  FIG.  6-38;  A/D  interconnect 
diagram,  FIG.  6-36) 

•  TO  33D7-38-77-28-1-1  (OAFI  test  summaries,  TABLES  2-10,  2-11-2-13,  2-26;  OAFI 
paragraph  references,  2-37,2-41, 2-45,  2-159) 


•  Voltage  measurements 

•  Test  control  operation 

•  Test  control  interpretation 

•  Signal  tracing 
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Table  6 

Sample  Problem  Statement. 


You  are  running  a  LRU-3  starting  at  test  segment  10,  entry  point  1,  the  power  check.  On  test 
segment  11,  test  0,  measurement  1,  the  CRT  console  shows  a  reading  of  9.99999+37  ohms  when 
performing  a  UUT  identifier  check.  The  CRT  displays  the  following  message: 

P  141040  TOS  11D0R24 
TSGO  11 

H  1  9.99999+37  OHMS  4.90364+02  4.03709+02  UUTIDENT 

CHECK  INTERFACE  HARDWARE 

TEST  PROGRAM,  UUT  P/N  AND/OR  REPLACE  2R01 

END  OF  TEST 


details  of  the  problem  (i.e.,  which  parts  of  the  system  are  relevant  to  consider  in  the  problem  and 
which  parts  are  not,  how  the  problem  affects  relevant  system  components,  how  specific 
hypotheses  might  be  confirmed  or  disconfirmed,  and  so  forth).  The  analyst's  familiarity  with  the 
problem  enables  her/him  to  ask  informed  questions  in  later  stages  when  other  individuals  are 
attempting  to  solve  the  problem.  The  analyst's  understanding  of  the  problem  greatly  influences 
the  accuracy  and  completeness  of  the  PARI  solution  traces  collected  later. 

Conceptually,  the  preparatory  PARI  interview  is  motivated  by  the  same  principles  that 
influence  the  basic  interview  that  occurs  in  later  stages.  Specifically,  with  the  PARI  approach,  an 
empirically  derived  structure  is  imposed  on  the  diagnostic  problem  solving  process  to  elicit 
Actions,  Precursors  to  Actions,  Results,  and  Interpretations  of  Results.  This  structure  assumes 
that  a  certain  pattern  of  regularity  exists  in  human  reasoning  about  complex  systems.  The 
expected  regularity  is  related  to  Clancey's  (1986)  assertion  that  system  diagnosis  is  not  simply  the 
name  of  a  disease,  i.e.,  it  is  not  a  static  product.  Rather,  it  is  a  dynamic,  self-improving  argument. 
The  evolving  argument  (diagnosis)  systematically  relates  symptom  manifestations  to  responsible 
agents  in  cause  and  effect  terms.  The  PARI  approach  has  been  designed  to  capture  the  interactive 
steps  by  which  hypotheses  are  declared,  actions  are  taken  to  test  hypotheses,  symptoms  are 
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observed  as  results,  and  results  are  related  to  an  increasingly  delimited  set  of  responsible  agents  in 
cause  and  effect  terms.  In  short,  the  PARI  approach  is  intended  to  capture  the  evolving 
diagnostic  argument  in  which  symptoms  are  iteratively  related  to  possible  responsible  agents.  The 
mental  (PARI)  events  themselves  as  well  as  the  reasons  behind  the  selecting  and  sequencing  of  the 
PARI  solution  steps  are  made  explicit  in  the  interview. 

PARI  Interview  Procedure 


Overview.  The  interview  unfolds  according  to  the  structure  illustrated  in  Figure  4.  In  this 
figure,  questions  are  attached  to  the  interview  elements  to  illustrate  the  probes  that  we  have  found 
to  be  the  most  effective  for  use  by  task  analysts.  These  questions,  combined  with  the  PARI 
rehash  questions  (Figure  5),  are  designed  to  elicit  instances  of  the  multifaceted  knowledge 
structures  that  are  coordinated  during  complex  problem  solving.  Appendix  D  provides  an 
example  of  a  PARI  trace  generated  by  an  expert  solving  the  problem  described  in  Table  5.  The 
trace  contains  both  the  PARI  solution  and  the  elaborative  rehash  data.  Although  the  solution 
contains  a  large  number  of  technical  references,  the  interested  reader  will  find  the  example  helpful 
in  clarifying  both  the  structure  of  the  PARI  problem-solving  sessions,  and  the  nature  of  the 
resulting  data. 

Step  0.  The  interview  begins  as  the  expert  considers  the  information  given  in  the  problem 
statement  (context  and  symptoms)  (see  Table  6).  S/he  is  probed  by  the  analyst  for  an 
interpretation  of  the  presenting  symptoms.  This  initial  step  is  considered  Step  0  and  contains  only 
the  Interpretation  element  of  the  PARI  structure  since  the  preceding  Action  and  Result  are 
already  embedded  in  the  problem  statement.  The  expert  is  also  asked  to  generate  a  sketch  or 
(block-level)  diagram  to  illustrate  Step  0.  The  diagram  should  illustrate  what  is  happening  in  the 
equipment  as  established  by  the  problem  statement  and  by  inferences  made  by  the  expert  from  the 
symptom  and  contextual  information  that  is  initially  provided  in  the  problem  statement.  In 
general,  the  diagrams  are  very  useful  for  depicting  how  the  expert  envisions  system  operation  as 
the  diagnostic  argument  unfolds.  At  step  0  the  diagram  is  actually  the  expert's  representation  of 
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the  problem.  The  expert's  perspective  on  the  system  can  be  captured  as  s/he  mentally  parses  the 
equipment  into  meaningful  units.  During  the  course  of  the  interview,  the  diagrams  should 
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PROBLEM 
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STEPO 

INTERPRETATION 
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STEPN 


t  WHAT  DO  THE  INITIAL 

SYMPTOMS  TELL  YOU  ABOUT 
WHAT  IS  GOING  ON  IN  THIS 
PROBLEM? 

•  ABOUT  POSSIBLE  FAULT 

LOCATIONS? 

•  ABOUT  PARTS  OF  THE  SYSTEM 

THAT  CAN  BE  ELIMINATED 
FROM  CONSIDERATION? 

•  DRAW  ALL  ACTIVE  COMPONENTS. 

INCLUDE  THE  LARGER 
EQUIPMENT  UNIT  IN  WHICH 
A  COMPONENT  IS  EMBEDDED. 


N  =  N  + 1 


WHAT  WOULD  YOUR  FIRST 
(NEXT)  ACTION  BE  IN 
SOLVING  THIS  PROBLEM? 

•  ARE  THERE  PRIOR  STEPS 
YOU  WOULD  HAVE  TO  TAKE 
TO  PERFORM  THE  ACTION. 
EG.,  CONSULT  TECH  DATA, 
DISCONNECT  A  CABLE? 
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PRECURSOR 
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WHY  ARE  YOU  TAKING  THIS 
ACTION,  LE,  WHAT  IS 
YOUR  RfeASCN  IN  TERMS  OF 
EQUIPMENT  BEING 
TARGETED? 


L 


RESULT 
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•  WHAT  DOES  THE  RESULT 

TELL  YOU  REGARDING 
FAULT  LOCATION? 

•  CAN  YOU  ELIMINATE 

ANY  CIRCUITRY  FROM 
SUSPICION  AT  THIS  POINT? 
WHAT  COMPONENTS  DO 
YOU  NOW  SUSPECT? 

WHY? 

DRAW  A  DIAGRAM  THAT 
ILLUSTRATES  THE 
ACTION  JUST  TAKEN. 


Figure  4.  Problem  solving  sessions:  PARI  interview  structure. 

successively  reveal  the  component(s)  the  expert  targets  as  suspects  and  those  which  are 
eliminated  or  downgraded  as  suspects. 


Later  Steps.  For  each  of  the  succeeding  solution  steps,  the  expert  specifies  an  Action,  the 
cognitive  Precursor  to  the  action,  and  an  Interpretation  of  the  Result.  It  is  the  researcher's 
responsibility  to  elicit  from  the  solving  expert  a  clear  statement  of  each  of  these  events  as  well  as 
the  supporting  conceptual  knowledge,  or  reasons  behind  the  steps.  This  includes  the  diagrams 
representing  the  action  taken  at  each  step.  The  goal  is  to  document  the  PARI  structures  at  a  level 
of  detail  that  makes  clear  not  only  how  the  problem  was  solved  but  why  it  was  solved  in  the 
particular  way  it  was. 
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The  elements  of  a  PARI  structure  are  as  follows:  an  Action  can  be  thought  of  as  an 
operation  that  the  human  problem  solver  performs  on  the  system  (e.g.,  taking  a  measurement)  or 
as  an  operation  intended  to  collect  information  about  the  system  (e.g.,  tracing  schematics).  In 
either  case,  the  action  yields  a  specific  result  and  may  constitute  a  test  (or  partial  test)  of  an 
hypothesis.  Associated  with  each  action  is  a  Precursor  statement  that  describes  the  current 
hypothesis  or  the  focus  of  the  action  (in  terms  of  the  system  components  that  are  being  targeted), 
and  how  the  action  tests  the  hypothesis.  In  short,  the  precursor  provides  the  justification  or  goal 
of  the  action.  The  expert  also  generates  a  diagram  to  illustrate  system  components  that  are 
relevant  to  each  Action  and  Precursor  at  a  particular  step.  Given  that  in  the  preparatory  PARI 
interview  the  expert  knows  the  location  (or  source)  of  the  problem,  s/he  then  provides  the 
system's  response  to  the  action  performed  at  that  step  (the  Result!  and  produces  an  Interpretation 
of  the  result  in  terms  of  the  hypothesis  being  pursued.  The  Interpretation  should  reveal  what  the 
Result  tells  the  expert  about  the  location  (cause)  of  the  fault,  what  system  component(s)  can  be 
eliminated  or  downgraded  as  suspect  causes,  and  what  the  expert  now  considers  to  be  prime 
suspects  and  why.  The  step-by-step  elicitation  of  these  PARI  elements  is  continued  until  the 
problem  has  been  solved. 

PARI  Rehash  Procedures 


Once  the  initial  PARI  trace  has  been  recorded,  the  researcher  conducts  a  series  of 
"rehashes"  (as  part  of  the  preparatory  solution),  wherein  the  expert  verifies  and  elaborates  the 
initial  trace.  Altogether  there  are  five  rehashes  (see  Figure  5).  Examplar  rehash  data  are  shown  in 
Appendix  D. 

Rehash  #1 :  Verification  of  the  PARI  trace.  The  purpose  of  Rehash  #1  is  simply  to  verify 
the  solution  trace  by  having  the  expert  ensure  that  the  analyst's  documentation  is  accurate.  The 
researcher  reviews  the  transcript  with  the  expert  who  clarifies  any  ambiguities  and  makes  sure  that 
the  relevant  diagrams  are  accurately  drawn  and  labeled.  This  rehash  gives  the  researcher  the 
opportunity  to  ask  for  further  explanations  of  any  PARI  elements  that  are  not  fully  explicit  in  the 
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trace  and  to  become  better  prepared  for  presenting  the  problem  to  other  experts  and  less  skilled 
performers. 


REHASH  #1  •  IS  THE  SOLUTION  STEP  ACCURATELY 
RECORDED? 

•  WHAT  DID  YOU  NEED  TO  KNOW  TO  TAKE 
THIS  ACTION? 


REHASH  #2  •  what  other  results  could  have 

OCCURRED  AT  THIS  STEP? 

•  HOW  WOULD  YOU  HAVE  INTERPRETED 
THOSE  RESULTS?  (ASSUME  RESULTS 
FROM  ORIGINAL  SOLUTION  FOR 
EARLIER  STEPS.) 

REHASH  #3  *  WHAT  OTHER  ACTIONS  COULD  YOU  HAVE 
TAKEN  AT  THIS  STEP?  (ASSUME  ACTIONS 
FROM  ORIGINAL  SOLUTION  FOR  EARLIER 
STEPS.) 

•  RATE  EACH  ALTERNATIVE  ACTION  USING 
A  7-POINT  RATING  SCALE.  COMPARE 
EACH  ALTERNATIVE  TO  THE  ORIGINAL 
ACTION  AND  SAY  WHY  ONE  ACTION  IS 
PREFERRED  OVER  THE  OTHER. 


REHASH  #4  ♦  WHAT  other  precursors  could  you 
HAVE  CONSIDERED  AT  THIS  STEP? 
(ASSUME  PRECURSORS  FROM  ORIGINAL 
SOLUTION  FOR  EARLIER  STEPS.) 

•  RATE  EACH  PRECURSOR  USING  A  7-POINT 
RATING  SCALE.  COMPARE  EACH 
ALTERNATIVE  TO  THE  ORIGINAL 
PRECURSOR  AND  SAY  WHY  ONE  IS 
PREFERRED  OVER  THE  OTHER. 

REHASH  #5  •  GROUP  THE  ACTIONS  IN  THIS  LIST  THAT 
SEEM  TO  GO  TOGETHER,  ON  WHAT  BASIS 
DID  YOU  GROUP  EACH  SET  OF  ACTIONS? 


Figure  5.  PARI  rehash  sessions. 


The  remaining  rehashes  are  particularly  instrumental  in  enhancing  the  cognitive  model  and 
its  associated  instructional  power.  The  expert  is  probed  to  elicit  alternative  Results, 
Interpretations,  Actions,  and  Precursors  for  each  step  in  the  solution  and  further  to  evaluate  the 
merits  of  the  viable  alternative  Actions  and  Precursors.  In  addition,  the  expert  is  asked  to  group 
the  Action  elements  of  the  solution  steps  into  meaningful  clusters.  The  clusters  reveal  the 
problem  solver's  higher  level  plan  or  goal  structure. 


Rehash  #2:  Alternative  Results/Interpretations.  The  second  rehash  is  designed  to  elicit  the 
full  range  of  hypotheses  being  considered  by  the  expert  at  each  step.  For  each  step  in  the 
solution,  the  problem  solver  is  asked  to  state  other  possible  outcomes  of  the  action  taken  at  that 
step,  and  to  interpret  each  outcome.  Each  possible  outcome  (Result)  will  confirm  or  fail  to 
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confirm  some  member  of  the  hypothesis  set.  For  example,  suppose  the  original  result  of  the  Step 
shown  in  Figure  3  (an  ohms  measurement  through  the  LRU  ID  resistor)  was  that  a  "good"  signal 
was  present  (i.e.,  the  proper  resistance  was  being  read).  A  reasonable  Interpretation  of  that 
Result  would  be  that  there  is  good  continuity  through  the  resistor  and  therefore,  that  component 
is  assumed  good.  Accordingly,  the  LRU  containing  the  resistor  is  downgraded  as  a  possible 
source  of  the  problem  and  circuitry  before  and  after  the  resistor  is  upgraded.  Alternatively,  an 
infinite  resistance  reading  could  be  obtained  (indicating  discontinuity  in  the  path  through  the 
resistor),  in  which  case  the  above  interpretation  would  be  reversed.  There  are  of  course  other 
details  about  the  context  of  the  problem  such  as  which  devices  are  operative  in  a  particular  test, 
the  base  failure  rate  of  each  operative  device,  and  so  forth  that  skilled  technicians  use  to  fine  tune 
their  Interpretations  as  well  as  their  initial  hypothesis  set.  Troubleshooting  actions  allow  the 
expert  to  test  empirically  a  range  of  hypotheses  and  subsequently  to  use  results  to  mentally  adjust 
the  likelihood  of  each  suspected  cause. 

Rehash  #3 :  Alternative  Actions.  This  rehash  is  designed  to  elicit  and  evaluate  alternative 
procedures  (or  Actions)  to  investigate  the  equipment,  given  the  targets  (or  goals)  established  in 
each  Precursor  from  the  original  solution  trace.  As  ill-structured  problem  solving, 
troubleshooting  involves  selecting  a  particular  action  or  procedure  from  a  range  of  possible 
procedures  each  time  a  step  is  formulated.  In  this  rehash,  the  expert  is  asked  to  state  all  of  the 
procedures  (Actions)  s/he  would  consider  appropriate  for  pursuing  the  goal  stated  in  the 
Precursor.  For  example,  the  Precursor  (goal)  might  state  "I  want  to  see  if  component  X  on  the 
circuit  is  good".  One  procedure  (No.  1)  might  be  to  measure  the  output  of  component  X,  another 
(No.  2)  to  swap  component  X,  and  still  another  (No.  3)  might  be  to  determine  (through  a 
measurement)  if  component  X  is  receiving  the  correct  instructions  from  the  computer  to  set  up 
properly  to  process  the  incoming  signal.  Each  possible  procedure  has  specific  costs  and  benefits 
associated  with  it  that  contribute  to  the  underlying  conceptual  knowledge  used  by  skilled 
troubleshooters  in  this  kind  of  decision  making.  Eliciting  the  reasons  behind  the  decisions  is  thus 
very  important  strategic  knowledge  for  the  PARI  method  to  capture.  In  the  above  example,  for 
instance,  procedure  No.  1  (measuring  the  output  of  component  X)  might  be  justified  over  the 
others  because  it  yields  more  information  and  inflicts  less  damage  (wear  and  tear)  on  the 
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equipment  system  than  swapping.  Further,  if  swapping  is  the  selected  action  and  it  does  not  fix 
the  problem,  the  solver  cannot  localize  the  fault  to  a  smaller  segment  of  the  circuitry  as  is  possible 
with  procedure  No.  1.  Rehash  #3  is  concluded  by  having  the  expert  use  a  seven-point  rating  scale 
(ranging  from  "much  worse"  to  "much  better")  to  compare  each  alternative  procedure  that  s/he 
has  generated  to  the  original,  selected  procedure. 

Rehash  #4;  Alternative  Precursors.  Rehash  #4  is  designed  to  elicit  and  evaluate  alternative 
goals  or  Precursors  considered  by  the  expert  in  formulating  his/her  plan  for  investigating  the 
equipment.  As  ill-structured  problem  solving,  troubleshooting  requires  a  continuous  stream  of 
decisions  by  the  performer  to  determine  what  to  do  next  based  on  system  feedback  (Results). 

This  rehash  is  an  attempt  to  make  explicit  the  problem  solver's  reasons  for  focusing  on  one 
particular  target  (or  equipment  component)  to  the  exclusion  of  other  equipment  targets.  In  this 
rehash,  the  expert  is  asked  to  state  all  equipment  targets  (Precursors)  s/he  would  consider 
appropriate  given  the  previously  executed  steps.  For  example,  suppose  the  previous  steps  have 
reduced  the  suspicion  surrounding  component  X  (e.g.,  the  LRU  ID  resistor  shown  in  Figure  3.) 
Suppose  further  that  the  expert  decides  to  target  component  Z  (e.g.,  the  test  package)  in  the  next 
step.  Eliciting  the  reasons  behind  the  decision  to  target  component  Z  (to  the  exclusion  of  other 
possible  targets)  is  also  important  strategic  knowledge  for  the  PARI  method  to  capture.  In  the 
above  example,  for  instance,  targeting  the  test  package  over  components  within  the  test  station 
might  be  justified,  because  that  component  is  known  to  have  a  high  rate  of  failure,  or  because 
previous  steps  have  ruled  out  test  station  components.  Rehash  #4  is  concluded  by  having  the 
expert  use  the  same  seven-point  rating  scale  used  in  the  previous  rehash  to  compare  each 
alternative  Precursor  (equipment  target)  to  his/her  original  selected  Precursor. 

Rehash  #5:  Grouped  Actions.  The  final  rehash  is  designed  to  elicit  higher  order  groupings 
of  actions  for  a  given  solution  after  the  solution  process  is  completed.  By  grouping  actions, 
solutions  can  be  analyzed  at  higher  levels  of  abstraction  where  commonalities  in  problem  solving 
across  experts  as  well  as  across  problems  are  most  likely  to  become  apparent.  Procedurally,  each 
expert  is  presented  a  list  that  contains  the  actions  of  the  original  solution  trace  s/he  produced. 

The  expert  is  then  asked  to  group  the  actions  that  seem  to  go  together  and  to  explain  the  basis  for 
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the  resultant  groupings.  The  groupings  tend  to  reveal  the  larger  chunks  of  the  problem  solver's 
plan  for  investigating  the  equipment  and  reflect  the  underlying  device  models  of  the  equipment 
that  give  rise  to  the  plan. 

Result  of  Stage  V 

The  result  of  Stage  V  is  the  set  of  solutions  (one  solution  per  problem)  generated  by  the 
experts  who  developed  the  problems.  In  the  next  stage,  experts  present  the  problems  they  have 
developed  to  each  of  the  other  experts  to  solve. 

In  the  discussion  of  Stage  VI  that  follows,  we  examine  the  data  produced  by  the  problem 
solving  interview/rehashes  just  described  and  explain  how  such  data  contribute  to  a  cognitive 
model  of  performance  to  inform  instruction. 

Stage  VI:  Generation  of  Expert  Solutions 


Goal  and  Rational 


As  stated  earlier,  the  cornerstone  of  the  PARI  method  is  the  generation  of  problem  solutions 
by  experts  who  are  naive  to  the  source  of  the  problem.  In  this  stage,  pairs  of  experts  come 
together  with  the  problems  (and  anticipated  solutions)  they  developed  independently  in  Stage  V. 
In  the  larger  data  collection  context,  solutions  are  elicited  from  experts  prior  to  conducting 
interviews  with  intermediate  and  novice  technicians.  There  are  several  reasons  for  this  sequence. 
First,  the  more  expertise  the  problem  solver  has,  the  more  predictable  his  actions  are  to  another 
expert.  The  set  of  actions  that  must  be  anticipated  by  the  problem  poser  is  therefore  rather 
constrained  for  expert  PARI  interviews.  The  experience  of  posing  one's  problem  to  a 
contemporary  is  nonetheless  useful  preparation  for  presenting  the  problem  to  less-skilled 
performers. 

By  conducting  the  initial  interviews  with  experts,  the  researcher/analyst  likewise  gains 
valuable  experience  at  the  outset  about  problem  solving  from  the  most  capable  and  articulate 
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performers.  Less  skilled  performers  may  not  be  able  to  articulate  the  reasons  for  their  actions  or 
may  even  choose  actions  that  have  no  logical  basis.  Cohesive  PARI  data  are  therefore  more 
difficult  to  collect  as  skill  level  decreases,  and  this  can  cause  confusion  for  the  analyst  who  may 
have  only  minimal  knowledge  of  the  domain. 

Procedure 


The  PARI  interview  with  expert  dyads  is  conducted  by  the  task  analyst  in  much  the  same 
manner  as  described  in  Stage  V;  therefore,  we  give  limited  attention  to  the  procedural  aspects  of 
Stage  VI  in  this  section.  Instead,  our  emphasis  is  on  how  the  various  elements  of  the  PARI 
structure  inform  a  cognitive  model  of  skilled  performance,  the  building  of  which  is  the  top-level 
goal  of  the  cognitive  task  analysis.  There  are,  nonetheless,  several  noteworthy  procedural 
activities  that  are  specific  to  this  stage.  They  will  be  addressed  first. 

The  first  procedural  feature  of  note  concerns  how  the  roles  of  the  two  experts  are  defined 
for  this  stage.  Expert  1  poses  the  problem  s/he  has  developed  to  Expert  2  by  presenting  the 
problem  statement  (from  Stage  V;  see  Table  6)  and  by  giving  Results  to  each  Action  taken  by  the 
second  expert,  or  solver  (Expert  2).  The  Results  are  stated  as  outcomes  that  would  be  achieved  if 
the  solver  were  actually  performing  the  stated  actions  on  the  real  equipment,  vs  verbally  solving 
the  problem  in  a  PARI  interview.  Expert  1  (the  problem  poser)  is  directed  to  ensure  that  the 
information  given  to  the  problem  solver  as  Results  is  as  close  as  possible  to  the  readings,  features, 
and  displays  of  the  equipment  operating  in  the  real  job  environment.  To  illustrate  the  point, 
changes  in  display  readings  which  would  be  immediately  visible  if  the  problem  solver  were 
interacting  with  the  actual  system  might  influence  performance  in  the  real  job  environment  and 
should  thus  be  provided  as  Results  in  the  simulated  PARI  situation. 

Expert  1  is  also  instructed  to  perform  "reality  checks"  during  the  PARI  interview  by 
questioning  any  Actions  of  the  problem  solver  that  would  not  be  normally  taken  in  the  real  task 
environment.  Similarly,  the  problem  poser  is  directed  to  query  Interpretations  (of  Results)  by  the 
solver  that  may  be  an  artifact  of  the  simulated  PARI  situation. 
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Expert  2  (the  solver)  is  instructed  to  show  normal  shop  behavior,  to  visualize  the  physical 
equipment  in  "the  mind's  eye"  so  that  salient  physical  features  of  the  equipment  are  noticed,  and  to 
think  out  loud,  clearly  stating  troubleshooting  Actions  and  surrounding  events.  The  solver  is  also 
asked  to  sketch  a  block  diagram  to  illustrate  each  step  in  the  PARI  sequence. 

Results  of  Stage  VI 

We  turn  now  to  an  examination  of  how  the  PARI  data  inform  the  model  of  skilled  problem 
solving  described  in  Section  I  (see  Figure  1).  Multiple  expert  solutions  to  a  variety  of  problems 
provide  the  primitive  pieces  of  knowledge  and  reasoning  processes  that  collectively  constitute  the 
experts'  coordinated  knowledge  structures.  We  do  not  believe  that  the  PARI  procedure  itself 
biases  the  data  in  favor  of  one  performance  model  over  another,  that  is,  the  PARI  structure  does 
not  determine  a  priori  the  types  of  results  that  can  be  obtained.  This  assertion  is  supported  by 
comparisons  of  PARI  data  obtained  from  experts  to  that  obtained  from  less-skilled  performers; 
the  data  reveal  differences  not  only  in  the  content  and  structure  of  their  knowledge,  but  also  in  the 
coordination  of  different  knowledge  sources  as  well.  In  the  following  discussion,  we  rely  heavily 
on  our  own  data  from  avionics  technicians  to  demonstrate  the  utility  of  the  PARI  procedure  in 
modelling  cognitive  performances,  both  skilled  and  unskilled. 

System  Knowledge  Structures.  In  the  BJS  cognitive  skills  architecture  (Figure  1), 
knowledge  of  how  the  system  works  is  assumed  to  be  central  to  both  procedural  and  strategic 
knowledge.  System  knowledge  both  drives  strategic  decisions  and  allows  the  problem  solver  to 
deploy,  tailor,  or  infer  appropriate  procedures  to  use  in  efficiently  investigating  the  equipment. 
Knowledge  of  equipment  systems  is  captured  in  several  ways  by  the  PARI  task  analysis.  First, 
experts'  descriptions  of  the  system  which  they  generate  during  the  equipment  orientation  phase  of 
Stage  I  specify  the  content  of  the  general  system  model  possessed  by  the  expert.  During  problem 
solving,  pieces  of  this  general  system  model  are  deployed  and  organized  to  form  a  series  of  device 
models  that  are  specific  to  the  problem  being  diagnosed.  Each  model  is  used  as  a  basis  for 
generating  hypotheses  concerning  the  problem  source  (or  cause)  as  stated  in  the  Precursors,  and 
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to  construct  tests  of  those  hypotheses  as  specified  in  the  Actions.  At  each  step  of  the  solution,  the 
expert  problem  solver  interprets  the  result  in  terms  of  the  current  device  model  and  then  updates 
the  model  to  reflect  new  information. 

The  system  knowledge  that  is  accessed  to  form  a  problem-specific  device  model  is  primarily 
captured  in  PARI  interviews  via  the  diagram  generated  at  each  step  of  the  solution  and  in  the 
Interpretation  element  that  integrates  new  information  into  the  diagnostic  argument.  The  diagram 
represents  both  an  instantiation  of  general  system  knowledge  and  a  special  purpose  representation 
of  the  device  —  particularized  to  the  symptoms  and  context  of  the  problem.  An  example  taken 
from  an  avionics  expert's  solution  demonstrates  how  general  system  knowledge  is  used  to 
generate  specific  device  models.  (The  entire  solution  trace  for  this  example  and  the  associated 
diagrams  and  rehash  data  are  contained  in  Appendix  D.) 

In  this  problem,  a  "test  station"  is  being  used  to  test  a  piece  of  jet  equipment  called  an 
"LRU,"  or  line  replaceable  unit.  Figure  6  shows  a  diagram  of  the  general  equipment  configuration 
during  LRU  testing.  An  LRU  is  a  black  box  component  that  has  been  removed  from  the  aircraft 
because  of  a  suspected  fault.  It  is  connected  to  the  test  equipment,  or  test  station,  via  a  "test 
package"  consisting  of  a  cable  and  an  adapter  which  serves  as  an  interface  between  the  LRU  and 
the  station.  In  testing  an  LRU,  a  stimulus  device  in  the  test  station  generates  a  signal  which  is 
sent  to  a  routing  device  and  then  relayed  to  the  LRU.  The  LRU  generates  a  response  to  the 
stimulus  which  is  then  sent  back  to  the  routing  device  in  the  test  station  and  relayed  to  a 
measurement  device.  In  this  example  problem,  the  LRU  is  being  tested  with  a  computerized 
sequence  of  procedures  when  the  test  on  the  LRU  identification  resistor  fails.  The  problem  is  to 
isolate  the  fault  that  caused  this  test  to  fail.  Diagrams  and  PARI  Interpretation  elements  reveal 
the  system  knowledge  that  is  deployed  in  the  fault  isolation  (problem  solving)  process,  as 
described  below. 

Diagrams.  Information  in  the  problem  statement  is  used  by  the  expert  to  construct  an  initial 
(mental)  representation  of  the  problem,  which  is  then  depicted  in  a  diagram  produced  after  the 
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Step  0  Interpretation  is  made.  The  diagram  and  interpretation  are  in  response  to  probes  from  the 
task  analyst  such  as,  "What  is  going  on  in  this  failed  test?  What  do  the  symptoms  tell  you  about 


TEST  STATION 


Figure  6.  General  equipment  configuration  during  LRU  testing. 

the  possible  causes  of  the  problem?  What  parts  of  the  equipment  will  you  initially  target?"  Figure 
7  shows  the  Step  0  diagram  produced  by  expert  RK.  Notice  the  close  parallels  between  the 
general  equipment  diagram  in  Figure  6  and  this  specific  diagram.  Both  diagrams  incorporate  the 
test  station,  test  package  and  LRU;  however,  in  the  generic  diagram  (Figure  6),  components 
within  the  test  station  are  given  generic  labels,  i.e.,  stimulus,  measurement,  and  routing 
components.  RK's  Step  0  diagram  provides  labels  to  designate  specifically  the  active  components 
in  this  particular  test,  i.e.,  to  illustrate  what  was  going  on  when  the  particular  fail  occurred.  In 
this  test,  both  stimulus  and  measurement  functions  are  performed  by  the  digital  multifunction 
meter  (or  DMM),  and  so  that  labeled  component  in  Figure  7  serves  both  functions.  The  specific 
routing  component  here  is  the  interface  chassis.  RK's  problem-specific  diagram  also  designates  a 
particular  LRU,  "LRU-3",  and  a  particular  interface  adapter,  "LRU-3  I/A"  (indicating  that  this 
adapter  is  used  specifically  in  testing  the  LRU-3).  The  expert's  Step  O  diagram  thus  constitutes 
an  instantiation  of  a  schema  of  the  general  equipment  configuration. 
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Figure  7.  R.K.’s  Step  0  diagram  showing  equipment  configuration  for  a  particular  LRU  test. 

Interpretation  Elements.  System  knowledge  is  also  elicited  during  the  Interpretations  of 
PARI  steps.  Interestingly,  the  Interpretations  reveal  the  close  interplay  between  system  (or  how- 
it-works)  knowledge  and  strategic  (or  how-to-decide-what-to-do-and-when)  knowledge.  To 
illustrate,  the  Step  0  Interpretation  for  the  diagram  depicted  in  Figure  7  included  the  following 
observations:  "This  test  checks  the  LRU  ID  resistor,  and  the  DMM  (digital  multifunction  meter) 
is  being  used  as  the  measurement  device;  I  will  initially  focus  on  the  LRU  because  the  LRU  ID 
resistor  may  actually  be  bad  as  the  failed  test  indicates,  or  a  pushed  pin  in  the  test  package  could 
have  caused  the  fail;  since  a  pushed  pin  is  more  likely  than  a  bad  ED  resistor,  the  problem  is  more 
likely  to  be  in  the  test  package  than  in  the  LRU.  I  won't  focus  on  the  test  station  initially  since 
troubleshooting  the  station  is  more  difficult;  I'll  rule  out  the  easier  components  first." 

The  strategic  knowledge  revealed  here  includes  the  following:  the  first  decision  was  to 
identify  the  components  of  the  system  that  were  active  when  the  test  failed  ("This  test  checks  the 
LRU  ID  resistor  and  the  DMM  is  used  as  the  measurement  device.")  This  decision  significantly 
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constrains  the  equipment  to  be  initially  searched.  Further,  cost-benefit  reasons  are  revealed  that 
guide  the  sequencing  of  solution  steps:  "Since  a  pushed  pin  is  more  likely  than  a  bad  ID  resistor, 
the  problem  is  more  likely  to  be  in  the  test  package  than  in  the  LRU.  I  won't  focus  on  the  test 
station  initially  since  troubleshooting  the  station  is  more  difficult;  I'll  rule  out  the  easier 
components  first."  Notice  that  these  reasons  are  tied  directly  to  knowledge  of  the  equipment 
system,  thus  requiring  access  to  detailed  knowledge  of  the  general  system. 

The  coordination  of  particularized  system  and  strategic  knowledge  by  this  expert  is  further 
highlighted  when  compared  with  the  types  of  Interpretations  produced  by  novices.  Due  to 
impoverished  system  knowledge,  novices  appear  to  default  to  general  or  weak  problem  solving 
methods  as  strategy  (e.g.,  find  someone  or  something  that  will  tell  me  what  to  do  next).  They 
show  little  capability  to  infer  either  a  focus  for  their  investigation  or  problem-specific  diagnostic 
procedures  from  the  system  description  provided  in  the  problem  statement. 

Knowledge  of  Procedures  and  Operations 

Using  the  same  illustrative  problem  described  above,  we  now  consider  how  the  PARI  data 
yield  information  about  the  performer's  knowledge  of  troubleshooting  procedures/operations. 
Coordination  between  procedural  knowledge  and  system  knowledge  is  again  the  most  salient 
finding  in  our  own  PARI  studies.  The  instances  of  procedural  know-how  tend  to  be  contained  in 
all  four  PARI  elements  and  the  alternative  actions. 

Actions.  Primarily,  procedural  knowledge  is  revealed  in  the  Action  statements  of  PARI 
solutions,  as  well  as  in  the  Alternative  Actions  that  result  from  Rehash  #3.  When  abstracted  from 
the  solutions  of  multiple  experts,  these  Action  instances  can  be  grouped  and  made  general  to 
provide  an  inventory  of  the  cognitive  procedural  skills  required  for  task  performance. 
Interestingly,  in  our  work,  experts  and  novices  do  not  appear  to  differ  substantially  in  then- 
knowledge  of  troubleshooting  procedures  (e.g.,  taking  measurements,  swapping  components, 
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checking  connections,  running  computer  diagnostics,  etc.).8  In  most  cases  the  novices  we  study 
have  already  acquired  some  procedural  skills  in  their  initial  technical  training  prior  to  reporting  to 
their  first  job  assignment.  This  is  not  surprising  since  the  execution  of  procedures  is  the  most 
readily  observable  aspect  of  task  performance  and  therefore  represents  an  obvious  target  for 
training.  However,  despite  some  knowledge  of  procedures  as  the  "tools  of  troubleshooting," 
novices  generally  possess  these  actions  detached  from  their  conditions  of  use.  They  may  know 
how  to  execute  a  procedure  but  frequently  fail  to  produce  the  specific  variation  of  the  procedure 
at  the  appropriate  time.  This  deficiency  is  revealed  in  Precursor  statements,  where  performers  are 
probed  for  the  reasons  for  their  actions. 

Precursors.  What  novices  appear  to  be  lacking  is  (strategic)  knowledge  that  empowers 
them  to  select  optimal  procedures  that  enable  meaningful  progress  during  diagnoses.  This 
conclusion  is  supported  by  an  examination  of  the  reasons  that  experts  and  novices  give  for  their 
Actions  as  Precursors  or  goals  being  pursued.  Whereas  experts  tend  to  select  an  Action  driven  by 
a  goal  that  in  effect  represents  a  specific  hypothesis  about  the  malfunction,  novices  tend  to  be 
much  less  hypothesis  driven.  For  example,  an  expert  who  elects  to  measure  the  path  through  the 
LRU  that  is  active  for  the  ID  resistor  test  (see  Figure  7)  would  be  likely  to  provide  as  a 
supporting  reason  (i.e..  Precursor  or  goal),  "I'm  taking  this  measurement  because  I  want  to 
eliminate  the  LRU  as  the  source  of  the  problem."  It  is  important  to  note  that  the  measurement 
procedure  itself  requires  prerequisite  procedural  skills  since  the  performer  must  be  able  to  identify 
the  test  points  from  technical  documentation  as  well  as  know  what  kind  of  measurement  (ohms 
vs.  AC  voltage  vs.  DC  voltage)  is  appropriate  to  test  the  stated  hypothesis.  This  cluster  of 
associated  requirements  is  often  daunting  to  novices.  They  typically  have  less-focused  goals  as 
hypotheses  and  only  weak  support  for  their  actions.  For  example,  a  novice  Action  for  this  same 
problem  may  be  to  replace  the  ID  resistor.  The  associated  Precursor  or  reason  may  reflect  no 
hypothesis  at  all  and  no  expectations  of  the  meaningfulness  of  the  Result.  Rather,  the  novice's 
Precursor  is  likely  to  be  "I'm  doing  this  because  the  computerized  test  tells  me  to  swap  the  resistor 
as  a  remedial  action."  Rather  than  identifying  the  information  they  need  (or  specifying  an 


8  The  notable  differences  concern  the  well-tuned  or  strong  procedures  displayed  by  experts  versus 
the  general,  domain  independent,  or  weak  procedures  often  employed  by  novices. 
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hypothesis)  and  then  selecting  a  procedure  that  will  serve  that  goal,  novices  often  choose 
procedures  based  on  instructions  from  an  outside  source,  or  because  they  think  the  procedure 
might  provide  some  useful  information.  They  are  often  uncertain  about  what  they  expect  to  learn, 
however. 

It  should  be  noted  that  novices'  reliance  on  procedures  provided  by  an  external  source  is  not 
necessarily  maladaptive  or  ill  advised.  The  external  support  can  provide  scaffolding  that  assists 
learning  as  long  as  the  reasons  behind  the  procedures/decisions  are  made  clear  in  the  external 
information  source.  Further,  if  executing  prescribed  procedures  gives  novices  valid  experiences 
interacting  with  the  system  and  observing  its  behavior,  then  these  experiences  become  the  basis 
for  developing  a  general  model  of  how  the  system  works.  Providing  reasons  for  the  procedures 
always  enriches  those  learning  experiences. 


Results/Interpretations.  Knowledge  of  problem  solving  procedures  is  also  revealed  in  the 
performer's  Interpretation  of  the  Result  of  the  procedure  (Action).  Without  well-developed 
knowledge  of  how  a  procedure  can  be  used  to  investigate  the  equipment,  erroneous 
Interpretations  of  Results  can  be  made.  This  phenomenon  can  be  illustrated  by  an  example  taken 
from  two  solutions  to  the  problem  described  earlier.  (The  reader  may  refer  to  Figure  7  which 
shows  the  equipment  component  targeted  by  the  action  taken  in  this  example.)  In  this  problem, 
both  an  expert  and  a  novice  chose  to  investigate  the  test  package  by  running  computer  diagnostics 
on  it.  Each  technician  was  told,  as  the  Result,  that  the  voltage  checks  passed  while  the  ohms 
checks  failed.  The  novice  interpreted  the  failed  ohms  checks  to  mean  that  the  test  package  was 
bad.  He  reasoned  that  since  the  purpose  of  a  diagnostic  program  is  to  detect  malfunctions  in  the 
test  package,  the  ohms  checks  must  have  failed  because  of  bad  components  in  the  test  package. 
This,  however,  was  not  the  source  of  the  problem.  The  novice's  failure  to  understand  how  the 
test  package  was  tested  and  the  role  of  other  test  station  components  in  running  the  diagnostics 
led  him  to  the  erroneous  conclusion  that  the  test  package  was  bad. 

By  contrast,  expert  RK's  Interpretation  of  this  result  revealed  inferences  that  were  clearly 
based  on  his  knowledge  of  how  the  test  package  diagnostic  procedure  works  and  how  the 
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components  of  the  equipment  are  interconnected.  His  Precursor  stated  that  this  procedure  checks 
the  internal  circuitry  of  the  interface  adapter  and  that  he  thought  this  circuitry  consisted  of  straight 
wires  through  the  interface.  When  all  of  the  ohms  checks  failed,  he  correctly  concluded  that  the 
problem  was  in  the  test  station  "because  there  is  no  circuitry  in  the  test  package  that  is  common  to 
all  of  the  ohms  checks."  He  knew  that  these  ohms  checks  were  simply  a  series  of  continuity 
checks  through  individual  wires  in  the  interface  adapter  and  that  the  likelihood  of  multiple  broken 
wires  was  low.  In  short,  he  understood  how  the  procedures  worked  and  what  conclusions  to 
draw  from  its  possible  Results.  The  more  probable  cause  was  a  failure  in  the  test  station  that 
produced  the  observed  effect  in  the  test  package  diagnostic  procedure.  The  importance  of  a  well- 
integrated  device  model  is  clearly  integral  to  mindfully  using  and  interpreting  procedures,  as 
illustrated  in  this  example. 

These  illustrative  data  support  the  view  that  knowing  how  to  execute  basic  procedures  is 
insufficient  for  skilled  diagnosis  and  fails  to  discriminate  between  expert  and  novice  problem 
solvers.  While  the  procedure  employed  by  both  expert  and  novice  was  identical,  their  reasons  for 
using  that  procedure  and  especially  their  interpretations  of  its  outcome  were  the  distinguishing 
characteristics  of  the  two  performances.  The  PARI  procedure  captured  the  telling  details 
surrounding  the  use  of  the  procedure,  including  critical  performance  components  such  as  system 
and  strategic  knowledge. 

Strategic  Knowledge.  While  the  presence  of  the  final  cognitive  skill  —strategic  knowledge- 
has  been  noted  in  our  earlier  treatments  of  PARI  data,  it  is  appropriate  to  conclude  this  major 
section  with  a  discussion  of  how  strategic  decision  making  is  elicited  in  PARI  interviews.  First, 
the  overall  plan  or  goal  structure  of  the  solution  is  captured  in  the  groupings  of  actions  obtained 
in  the  fifth  rehash.  In  addition,  the  selection  and  sequencing  of  solution  steps  is  at  the  core  of  ill- 
structured  problem  solving  and  thus  strategic  decisions  can  be  accurately  regarded  as  the  glue  of 
the  solution  process.  The  nature  of  those  decisions  is  perhaps  best  revealed  in  the  three  rehash 
sessions  where  performers  are  asked  to  produce  and  evaluate  alternative  Actions,  Precursors,  and 
Results/Interpretations.  For  example,  when  selecting  a  Precursor  or  Action,  the  expert  considers 
the  amount  of  information  that  will  be  gained  to  further  constrain  the  problem  space  and  make 
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progress  toward  restoring  equipment  functioning.  The  selection  of  Precursors  and  Actions  is 
therefore  fundamentally  tied  to  the  expert's  anticipation  of  alternative  Results  and  the  benefit 
ascribed  to  achieving  those  Results.  Major  factors  considered  by  the  expert  in  strategic  decision 
making  include  costs  such  as  mental  and  physical  effort,  time,  danger,  and  equipment  replacement 
costs.  Benefits  to  be  optimized  include  constraining  the  possible  sources  of  malfunction  and 
making  progress  toward  restored  equipment  functioning. 

Alternative  Actions  and  Precursors.  Knowledge  of  strategic  decisions  is  captured  in 
alternative  precursors  and  alternative  actions  rehashes  (Rehashes  3  and  4).  Unlike  novices, 
experts  have  multiple  ways  of  approaching  problems  which  allow  them  to  choose  among 
alternative  equipment  targets  (Precursors)  and  alternative  investigatory  procedures  (Actions). 
Both  the  contextual  surround  of  the  problem  and  information  gained  at  previous  solution  steps 
influence  the  selection  process.  In  Rehash  3,  alternative  actions  are  elicited  from  experts  to  make 
explicit  the  range  of  procedures  considered  appropriate  to  achieve  a  particular  goal.  The 
procedural  options  being  considered  by  the  problem  solver  are  thereby  revealed.  Similarly,  in 
Rehash  4,  alternative  precursors  are  elicited  to  establish  the  viable  equipment  targets  considered 
as  goals  for  investigation. 

For  example,  in  the  problem  described  earlier,  a  technician  might  have  several  procedures 
(actions)  available  to  determine  whether  the  LRU  ID  resistor  is  bad.  S/he  may  check  for 
continuity  through  the  resistor  by  taking  an  ohms  measurement,  replace  the  resistor,  or  swap  the 
LRU  and  then  rerun  the  failed  test  to  see  if  the  problem  has  been  fixed.  Similarly,  in  later  stages 
of  problem  solving,  the  expert  may  have  narrowed  down  the  suspect  causes  to  several  component 
devices  within  the  test  station.  S/he  may  target  the  measurement  device,  the  routing  device,  or 
even  the  computer  that  provides  input  to  the  other  station  components. 

After  listing  the  alternative  Actions  and  Precursors  in  rehash  sessions,  experts  are  asked  to 
compare  the  original,  selected  Precursor  and  Action  to  the  listed  alternatives  in  an  informal  "cost- 
benefit  analysis."  The  relative  merits  of  each  option  are  weighed  by  the  problem  solver  who 
explains  the  reasons  why  one  Precursor  or  Action  should  or  should  not  be  chosen  over  others. 
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We  noted  earlier  that  in  the  avionics  diagnosis  domain,  these  decisions  appear  to  be  based 
on  a  strategy  that  attempts  to  maximize  the  information  gained  at  a  particular  solution  step  while 
minimizing  the  physical  effort,  mental  effort,  time,  danger,  and  cost  associated  with  carrying  out 
that  step.  For  any  given  strategic  decision,  estimates  of  these  factors  in  turn  depend  on  the 
individual  problem  solver's  knowledge  of  the  system.  Because  system  knowledge  varies  across 
technicians,  their  decisions  concerning  the  most  efficient  steps  to  take  also  vary.  One  expert  may 
choose  to  investigate  a  component  by  measuring  its  output  while  a  second  expert  may  choose  to 
run  computer  diagnostics  on  it;  although  the  latter  procedure  might  require  less  physical  effort,  it 
might  also  require  more  mental  effort  if  the  technician  lacks  the  system  knowledge  needed  to 
determine  what  the  diagnostics  are  doing  and  to  interpret  the  Results  correctly.  The  weight  given 
to  these  factors  by  a  single  expert  may  also  vary  across  situations.  In  each  situation,  the  expert 
must  identify  the  tradeoffs  among  the  various  alternatives  available  in  order  to  make  a  good 
decision.  However,  by  requiring  technicians  to  explain  what  makes  one  alternative  better  or  more 
efficient  than  another  in  each  specific  context,  the  cost-benefit  analysis  captures  what  is  common 
to  the  decisions  of  multiple  experts  (i.e.,  the  factors  that  underlie  these  decisions  in  general)  as 
well  as  what  is  unique  about  them. 

Alternative  Results  and  Interpretations.  Strategic  knowledge  is  also  revealed  when  experts 
are  asked  to  specify  alternative  results  that  may  have  occurred  in  response  to  a  selected  Action 
and  to  explain  how  such  an  outcome  would  have  been  interpreted.  Again  the  expert's  system 
knowledge  is  the  determining  factor  in  the  quality  of  strategic  Interpretations.  By  providing  a 
means  for  mentally  simulating  the  behavior  of  the  system,  a  robust  device  model  allows  the  expert 
to  anticipate  a  procedure's  possible  outcomes.  In  turn,  an  examination  of  the  set  of  possible 
Results  and  their  Interpretations  reveals  how  the  expert  distinguishes  empirically  between  multiple 
hypotheses  concerning  the  source  of  the  problem. 

For  example,  checking  a  signal  at  a  certain  point  along  the  signal  path  might  yield  one  of 
two  results:  the  signal  is  either  present  or  absent.  While  a  present  signal  indicates  that  the  fault  is 
located  downstream,  i.e.,  past  the  point  at  which  the  signal  was  measured,  an  absent  signal 


63 


isolates  the  problem  to  the  circuitry  upstream,  or  before  the  measurement  point.  This  "split-half1 
strategy  enables  the  technician  to  eliminate  a  large  portion  of  the  signal  path  from  further 
consideration,  but  does  not  allow  her  or  him  to  determine  whether  or  not  the  fault  is  located  in 
any  one  component.  Swapping,  on  the  other  hand,  will  allow  such  a  determination  to  be  made, 
but  is  usually  cost  effective  only  when  the  fault  has  been  isolated  with  a  high  degree  of  confidence 
to  the  component  to  be  swapped.  Procedures  are  thus  strategically  chosen  by  the  expert  to  allow 
particular  kinds  of  inferences  to  be  drawn  from  expected  results.  These  inferences  are  enabled  via 
an  internalized  device  model  and  are  articulated  in  the  Interpretation  element  and  the  alternative 
Result/Interpretation  rehash. 

Grouped  Actions.  Whereas  the  second,  third,  and  forth  rehashes  (alternative  Actions, 
Precursors,  and  Results/Interpretations)  capture  strategic  decisions  at  a  fairly  local  level,  the 
grouping  of  actions  reflects  the  overall  goal  structure  of  the  solution  and  thus  captures  more 
global  strategies.  In  generating  these  groupings,  the  problem  solver  is  free  to  group  actions  on 
whatever  basis  s/he  feels  makes  sense.  For  example,  actions  may  be  grouped  in  terms  of  the 
larger  functional  unit  being  investigated  in  a  set  of  actions,  or  in  terms  the  strategic  goal  being 
pursued.  In  general,  however,  we  have  found  in  our  studies  of  avionics  technicians  that  these 
groupings  are  closely  related  to  the  technician's  device  model  as  depicted  in  the  Step  0  diagram. 

For  instance,  in  expert  R.K.'s  solution  to  the  problem  described  earlier  (provided  in 
Appendix  D),  actions  are  grouped  in  terms  of  (a)  ruling  out  the  LRU,  (b)  ruling  out  the  test 
package,  (c)  correlating  the  results  of  multiple  tests  (to  narrow  down  the  targets  for  investigation 
within  the  test  station),  (d)  space  splitting  (within  the  test  station),  (e)  checking  the  instrument 
select  relays  (the  routing  drawer,  or  interface  chassis  circuitry  not  previously  eliminated)  and  (f) 
checking  the  path  from  the  interface  chassis  to  the  DMM.  R.K.'s  step  0  diagram  depicts  the 
functional  units  of  the  equipment  that  were  reflected  in  these  goals:  the  LRU  in  goal  a;  the  test 
package  in  goal  b;  the  interface  chassis  in  goal  e;  and  the  path  to  the  DMM  in  goal  f.  The 
groupings  also  reflect  strategic  goals:  ruling  out  components  external  to  the  test  station  first 
(goals  a  and  b);  correlating  the  results  of  multiple  tests  (goal  c);  and  space  splitting  (goal  d).  Thus 
we  find  that  the  device  model  deployed  to  represent  a  particular  problem  drives  the  overall  goal 
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structure  of  the  solution  and  provides  the  basis  for  high  level  strategic  decisions.  The  ability  to 
accurately  represent  the  problem  and  bring  relevant  facts  to  bear  on  its  solution  depends  on  the 
flexibility,  and  thus,  the  depth  and  quality  of  the  system  knowledge  base. 

Stage  VII:  Problem  Set  Review  by  Expert  Problem  Solvers 


Goal  and  Rational 


In  this  stage,  the  goal  is  to  obtain  judgments  on  the  goodness  of  the  problem  set  from  the 
experts  who  participated  in  the  PARI  workshop.  This  provides  an  opportunity  for  the  experts  to 
judge  on  the  basis  of  the  problem  solutions  they  have  seen  whether  the  problem  set  adequately 
tests  the  cognitive  skills  and  knowledge  required  for  expert  performance. 

Procedure 


Experts  are  asked  to  judge  whether  the  PARI  problems  constitute  a  representative  sample  of 
those  problems  seen  in  the  actual  job  environment,  whether  the  PARI  problems  adequately 
"exercise"  the  skills  and  knowledge  required  for  skilled  job  performance,  and  whether  the 
problems  have  strong  training  utility.  The  experts  are  also  asked  to  rank  order  the  problems  on 
level  of  difficulty  for  both  expert  and  novice  performers  and  to  rate  the  criticality  of  the  cognitive 
skills  required  for  problem  solutions.  Experts  are  asked  to  rate  each  identified  cognitive  skill  on 
three  dimensions:  usefulness  of  the  skill  in  problem  solution  and  in  overall  job  performance, 
learning  difficulty  associated  with  each  skill,  and  recommended  training  emphasis.  Appendix  E  is 
a  Problem  Set  Review  Questionnaire  used  in  BJS  studies  of  Avionics  job  specialties. 

Result  of  Stage  VII 

The  feedback  obtained  at  this  stage  is  used  for  multiple  purposes.  First,  judgments  about 
the  goodness  of  the  problem  set  guide  the  researcher  in  determining  whether  additional  problems 
should  be  generated  in  order  to  form  a  sound  instructional  base.  Because  an  instructional 
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assumption  underlying  our  program  is  that  expert  problem  solving  is  the  instructional  target,  it  is 
important  that  the  PARI  problems  as  a  group  require  a  representative  set  of  expert  skills  for 
solution.  Secondly,  judgments  regarding  problem  difficulty  and  skill  criticality  are  useful  in 
informing  both  later  PARI  interviews  with  less  skilled  performers  and  to  give  focus  and  order  to 
the  later  instructional  design  process. 


Stage  VIII:  Generation  of  Problem  Solutions  by 
Intermediate  and  Novice  Technicians 


Goal  and  Rationale 


Collecting  PARI  solutions  from  less-experienced  technicians  is,  in  most  respects,  identical  to 
collecting  data  from  experts  (see  Stages  V  &  VI).  The  primary  difference  is  that  rehashes  are 
restricted  to  the  first  two,  namely,  the  verification  rehash  and  the  alternative  results/interpretation 
rehash.  In  our  experience,  it  is  difficult  for  less-experienced  technicians  to  provide  alternative 
precursors  and  actions  because  they  require  the  articulation  of  alternative  hypotheses  and  test 
procedures  for  each  step  in  the  solution.  Generally,  less-skilled  performers  lack  such  a  range  of 
procedural  options  and  equipment  hypotheses.  Therefore,  frustration  occurs  when  they  are 
unable  to  answer  rehash  questions.  We  have  thus  eliminated  later  rehashes  during  novice 
interviews  to  reduce  the  potential  for  frustration  and  because  we  have  found  this  type  of  data 
from  less-experienced  personnel  to  be  uneven  and  uninformative. 

We  have  also  adopted  a  practice  of  presenting  problems  to  novices  in  order  of  increasing 
difficulty,  based  on  experts'  earlier  rankings  of  problem  difficulty  (for  novices).  The  goal  is  again 
to  minimize  novices'  frustration  and  maximize  the  quality  of  the  data  by  increasing  the  likelihood 
of  early  success  in  the  PARI  problem-solving  sessions. 
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Procedure 


ProceduraUy,  there  is  one  issue  that  is  unique  to  PARI  data  collection  with  less-skilled 
personnel,  and  that  concerns  how  to  continue  the  PARI  interview  if  the  problem  solver  seems  to 
have  hit  a  dead  end.  Since  our  purpose  in  conducting  interviews  with  less-skilled  performers  is 
not  to  see  how  many  problems  they  can  solve  independently,  but  instead  to  capture  the  content  of 
their  domain  knowledge,  our  practice  has  been  to  provide  assistance  to  compensate  for  gaps  in 
their  knowledge.  The  expectation  is  that  such  help  will  bridge  to  additional  knowledge  that  can 
be  revealed.  More  generally,  our  goal  is  to  develop  cognitive  models  that  characterize  the 
knowledge  structures  of  less-skilled  individuals  (at  their  particular  stage  of  development)  so  that 
instructional  decisions  such  as  learning  trajectories,  curriculum  sequencing,  and  so  forth  can  be 
better  informed.  Having  less  mature  cognitive  models  to  contrast  with  expert  models  in  effect 
highlights  salient  skill  differences  that  can  be  very  important  pedagogically. 

Toward  that  end,  we  have  adopted  the  practice  of  allowing  the  expert  who  is  posing  the 
problem  to  give  structured  assistance  to  the  solver  when  s/he  appears  to  be  no  longer  making 
progress  toward  a  solution.  This  practice  assumes,  of  course,  that  dead-end  situations  can  be 
defined  explicitly  enough  to  be  recognized  when  they  occur  and  that  principled  procedures  for 
giving  assistance  can  be  developed.  For  our  purposes,  dead-end  situations  are  those  in  which  a 
sequence  of  actions  (defined  as  x  number  of  solution  steps)  is  taken  without  deriving  information 
that  is  relevant  to  the  problem's  solution.  For  example,  in  the  avionics  domain,  a  dead-end  can 
occur  when  the  problem  solver  takes  a  series  of  steps  to  investigate  component(s)  that  are  not  on 
the  active  circuit  path,  when  the  technician  continues  to  investigate  a  component  that  has  already 
been  eliminated  as  the  source  of  the  problem,  or  when  the  technician  explicitly  states  that  s/he 
does  not  know  what  to  do  next.  Errors  such  as  misinterpreting  a  result  or  choosing  a  procedure 
that  does  not  test  the  stated  hypothesis  can  lead  to  dead-end  situations  as  well,  but  before  any 
help  is  given,  the  technician  is  given  several  steps  to  self-correct. 

In  order  to  provide  help  in  a  principled  way,  we  have  developed  several  types  of  hints  to 
guide  the  expert's  determination  of  what  information  to  give  a  problem  solver  when  a  dead-end 
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situation  as  defined  above  has  been  detected.  The  hint  structure  assumes  that  a  barrier  or  dead 
end  has  been  encountered  at  the  beginning  of  a  solution  step,  i.e.,  that  the  prior  step  has  been 
completed.  The  first  type  of  hint  given  in  this  situation  is  simply  a  review  of  all  previous  steps. 
Often  a  problem  solver  simply  loses  his/her  place  and  forgets  that  some  component  has  already 
been  tested  and  found  good.  As  a  consequence,  a  recapitulation  may  get  her/him  back  on  a  viable 
solution  path.  If  the  technician  is  still  having  problems  determining  what  to  do,  the  next  type  of 
hint  is  a  suggestion  concerning  which  component  might  be  targeted  for  investigation  that  is,  a 
Precursor  is  suggested.  Having  been  told  what  to  investigate,  the  technician  may  still  not  be  able 
to  state  an  action  that  would  test  that  component,  in  which  case,  an  appropriate  Action  is 
suggested  by  the  expert.  If  the  technician  is  unable  to  correctly  interpret  the  result  of  the 
suggested  action,  the  expert  interprets  it  for  him.  The  hint  structure  thus  corresponds  to  the 
PARI  elements  of  one  solution  step  (Precursor,  Action,  Result,  and  Interpretation).  If  the 
problem  solver  cannot  continue  independently  after  this  kind  of  assistance  on  one  step,  it  is 
assumed  that  the  technician's  knowledge  related  to  this  problem  has  been  exhausted  and  the 
interview  is  ended. 

Stage  IX:  Problem  Set  Review  by  Independent  (Advanced)  Experts 

The  purpose  of  the  final  stage,  Stage  IX,  is  to  have  the  problem  set  evaluated  by  senior 
experts  who  have  a  broader  experience  base  than  those  who  actually  participated  in  the  PARI 
workshop.  These  individuals  are  generally  in  management  positions  where  they  no  longer  work 
on  the  equipment  systems  being  studied.  However,  because  of  their  experience  at  a  variety  of  job 
sites,  they  are  in  a  position  to  identify  whether  there  are  site-specific  conditions  that  have 
inappropriately  influenced  the  problems  contained  in  the  problem  set.  Also,  they  are  asked  to 
review  the  accuracy  of  the  problem  and  evaluate  each  problem's  representativeness,  completeness 
and  utility  in  training.  Appendix  F  contains  a  questionnaire  designed  for  this  purpose  in  studies 
conducted  under  the  BJS  program.  This  final  evaluation  provides  an  independent  estimate  of  the 
goodness  of  the  problem  set  based  on  the  criteria  previously  used  in  generating  the  problems. 
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Summary 


The  preceding  description  of  the  PARI  data  collection  procedures  emphasizes  several 
themes  that  provide  the  basis  for  the  method's  design.  One  recurrent  theme  is  that  the  cognitive 
processes  and  structures  used  in  solving  problems  are  best  revealed  in  dyadic  interaction  during 
a  situated  problem-solving  task.  Thus,  experts  are  asked  to  generate  and  solve  realistic 
problems  in  a  setting  that  simulates  the  actual  task  conditions.  Second,  by  imposing  a  structure 
on  problem  solutions,  data  are  collected  systematically,  ensuring  that  the  knowledge  underlying 
a  problem's  solution  is  made  explicit  as  well  as  the  observable,  behavioral  solution  steps. 
Establishing  the  reasons  that  drive  particular  solution  steps  reveals  important  instructional  targets 
that  would  not  be  captured  if  these  reasons  were  not  systematically  accessed  or  probed.  Third,  in 
knowledge-rich  domains,  there  is  no  "preferred  solution"  to  any  given  problem  (i.e.  one  that  all 
experts  agree  on)  since  the  content  and  organization  of  experts'  knowledge  differ.  Thus,  the 
PARI  methodology  acknowledges  the  importance  of  input  from  multiple  experts.  Finally,  ill- 
structured  problem  solving  is  characterized  by  instability  in  the  task  environment  which  means 
that  no  single  solution  method  is  appropriate  for  solving  all  problems  under  all  conditions.  To 
establish  the  conditions  under  which  different  problem-solving  approaches  are  appropriate,  and 
what  influences  the  selection  of  a  problem-solving  strategy,  solutions  to  a  representative  sample 
of  problems  are  required. 
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IV.  UTILITY  OF  COGNITIVE  MODELS 


In  this  section  we  examine  the  application  of  the  PARI  methodology  as  a  knowledge 
representation  tool  to  aid  instructional  development  and  skill  assessment.  The  utility  of  PARI- 
generated  cognitive  models  in  practice-centered  instruction  is  actually  tied  to  the  nature  of  the 
targeted  skills.  Proficiency  in  modem  work  environments  often  requires  the  coordination  of 
multiple  types  and  levels  of  knowledge  under  diverse  conditions  to  pursue  various  interrelated 
goals.  As  a  result,  the  coupling  of  knowledge  to  goals  occurs  at  a  variety  of  levels  for  skilled 
performers,  suggesting  that  to  be  effective,  instruction  needs  to  be  informed  at  comparable  levels 
of  specificity  and  abstraction.  Cognitive  models  can  provide  the  necessary  detailed 
representations,  as  well  as  reveal  the  mechanisms  (strategies)  for  selecting  and  activating  the 
appropriate  knowledge.  The  models  can  ensure  that  the  interrelated  cognitive  components  that 
constitute  skilled  performance  are  treated  instructionally  and  that  the  forms  and  levels  of 
knowledge  targeted  by  instruction  can  meet  the  demands  imposed  by  actual  performance 
contexts. 

The  PARI  methodology  has  also  proven  useful  in  evaluating  training  developed  from  PARI- 
based  cognitive  models.  Since  PARI  problem  solving  sessions  are  situated  in  realistic  conditions 
that  simulate  the  task  environment  of  the  actual  job,  the  procedure  serves  as  a  practical 
assessment  tool,  yielding  valid  measures  of  technical  skill.  Performance  data  of  individuals  of 
unknown  proficiency  can  be  evaluated  against  the  models  which  in  turn,  are  based  on  the 
performance  of  individuals  at  known  levels  of  proficiency.  Thus,  the  PARI  methodology 
underlies  both  the  ability  to  model  or  define  the  criterion  performance,  and  the  ability  to  evaluate 
the  extent  to  which  an  individual  has  reached  the  criterion  performance  level. 

There  are  at  present  three  training  studies  associated  with  the  Air  Force  BJS  program  that 
provide  illustrative  examples  of  the  use  of  cognitive  models  from  PARI  data  to  inform  instruction 
and  skill  assessment.  Before  describing  those  studies,  we  will  quickly  review  the  principles 
underlying  the  PARI  procedures. 
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In  Section  I  of  this  paper  we  described  the  purpose  of  the  BJS  program  to  develop  an 
integrated  skill  analysis/instructional  development  technology  that  promotes  both  depth  and 
breadth  of  proficiency  in  the  maintenance  of  highly  complex  systems.  The  increasing 
sophistication  of  equipment  systems  used  in  virtually  every  technical  job  environment  often 
requires  a  deep  understanding  of  equipment  functionality  to  successfully  perform  complex 
diagnostic  tasks.  Second,  the  variety  of  equipment  systems  used  by  a  worker  throughout  a  career 
requires  adaptiveness,  such  that  mastering  the  functioning  of  one  system  should  foster  accelerated 
mastery  of  other  systems.  Developing  training  that  promotes  both  deep  and  broad  system 
understanding  is  critically  dependent  on  a  methodology  that  (1)  allows  the  criterion  performance 
(i.e.,  the  skilled,  adaptive  performance  of  the  expert)  to  be  modelled  in  detail,  and  (2)  that 
facilitates  comparative  analyses  of  performance  models  across  domains  as  well  as  across  differing 
levels  of  proficiency.  As  a  knowledge  representation  tool,  the  PARI  methodology  has  been 
developed  to  respond  to  our  needs  to  characterize  both  the  depth  and  breadth  of  technical 
expertise.  Further,  the  procedures  have  been  codified  for  use  by  nonscientific  personnel. 

The  following  description  of  training  studies  illustrate  in  general  how  the  output  of  a  PARI 
analysis  is  used  to  build  an  instructional  framework  and  ultimately,  curriculum  content  and 
method. 


PARI-Based  Training  Studies 


Proof-of-Principle  Study 

Rationale.  An  early  training  study  was  conducted  to  evaluate  the  use  of  performance 
models  derived  from  PARI  data  as  the  basis  of  instruction  (Gott  and  Pokomy,  1987).  In  this 
study,  the  general  approach  was  to  identify  differences  in  the  cognitive  performance  models 
(PARI  data)  for  expert,  intermediate,  and  novice  avionics  troubleshooters  and  to  design  a  training 
intervention  based  on  these  models.  The  expert  performance  models  provided  the  distal  goals  of 
the  training,  but  more  importantly  (for  this  short-term  undertaking)  the  less-than-expert 
performance  models  provided  the  basis  for  proximal  performance  goals,  thereby  informing 
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instructional  sequencing.  Collectively,  models  of  successive  approximations  of  expertise  provided 
a  learning  trajectory  as  a  skeletal  instructional  framework.  This  trajectory  enabled  us  to  determine 
whether  the  training  intervention  was  effective  in  moving  novice  technicians  toward  more  expert¬ 
like  troubleshooting  performance. 


Development  of  Cognitive  Models.  PARI  data  were  collected  from  a  range  of  F-15  avionics 
maintenance  personnel  so  that  expert,  intermediate  and  novice  performance  levels  could  be 
specified.  Cognitive  models  were  developed  by  extracting  from  all  PARI  traces  instances  of 
Precursors  and  Actions,  including  system  diagrams  and  supporting  reasons,  and  then  classifying 
instances  into  appropriate  cognitive  skill  categories.  For  example,  grouped  Action  instances 
yielded  procedural  categories  such  as  "visual  inspections",  "swapping",  "measurement  taking", 
and  "computer  control/software  interpretation."  Grouped  instances  of  Precursors  yielded  goal 
structure  categories  such  as  "verify  fail,"  "expand  information  on  probable  cause  of  fail,"  "test 
suspected  component's  inputs/outputs  to  locate  probable  cause  of  fail."  Table  7  provides 
examples  of  Action  and  Precursor  instances  extracted  from  the  PARI  data,  along  with  the 
categories  to  which  they  were  assigned. 


Table  7 

Grouping  and  classification  of  actions  and  precursors. 


ACTION  INSTANCE  PROCEDURAL  CATEGORY 


PRECURSOR  INSTANCE  GOAL  STRUCTURE  CATEGORY 


♦  Check  pins  on  test  package  Visual  inspections 

•  Check  fault  indicator  light 


•  Swap  Threat  Simulator  A5  card  Swapping 

•  Swap  card  N4A1  with  like  N4A3  card 


•  Run  diagnostics  on  high  frequency 
measurement  card  &  coax  switch 

•  Run  diagnostics  with  bit  dump 


Computer  Control/ 
Software  Interpretation 


•  Want  to  verify  5V  power  supply  Verify  Fail 

fail  not  a  fluke 

•  Want  to  verify  failed  diagnostic  not 
a  fluke 


•  Need  more  information  on  drawer 
serviceability 

•  Need  more  information  on  resources 
used  in  failed  test 

•  Want  to  trace  stimulus  input  to  get 
complete  routing 


Expand  Information  on 
Probable  Cause  of  Failure 
and  its  Inputs/Outputs 


•  Test  for  good  signal  out-  out  of  Measurement  Taking 

TP4  with  oscilloscope 

•  Ohms  check  between  J110  and  J4 
with  digital  multimeter  (DMM) 

•  Put  N4A1  on  extender  and  test  for 
18VDC  with  DMM 


•  Want  to  test  most  likely  suspect  on 
stimulus  path 

•  Want  to  check  other  cards  in  signal 
flow  path 

•  Want  to  check  for  good  input  signal 
at  N4A1 

•  Want  to  check  wiring  between  source 
of  signal  (N3A16)  and  N4A1 


Test  Inputs/Outputs  to 
Probable  Cause  of  Failure 
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The  procedural  categories,  e.g.,  measurement  taking,  led  to  fine-grained  explications  of  the 
procedure  in  a  narrative  writeup  called  a  skill  definition.  Skill  definitions  make  procedural  skills 
clear  enough  for  teaching  and  testing  purposes  by  providing  an  analysis  of  the  step-by-step 
subcomponents  of  the  skill  and  the  conditions  under  which  the  skill  is  activated.  The  coordination 
of  the  procedural  skill  with  system  and  strategic  knowledge  is  also  defined.  For  example, 
informed  decisions  in  selecting  among  procedures  such  as  visual  inspection,  swapping, 
measurement  taking,  or  using  computer  diagnostics  depend  on  the  current  goal  and  require  an 
evaluation  of  the  relative  costs  of  the  situation-based  alternatives  (time,  danger,  dollars,  mental 
and  physical  energy  required). 

Thus,  informed  procedural  decisions  require  strategic  knowledge.  Once  a  procedure  such  as 
measurement  taking  has  been  selected,  using  the  procedure  effectively  depends  on  system 
knowledge  such  as  knowing  how  to  read  and  interpret  the  measured  property.  Measurement 
taking  demands  such  as  these  and  the  knowledge  needed  to  assess  the  conditions  under  which  the 
skill  should  be  exercised  are  derivable  from  elements  in  the  PARI  nodes  and  rehashes.  The 
generation  of  skill  definitions  for  a  job  domain  serves  to  describe  in  detail  exactly  how  procedures 
are  executed,  how  systems  are  modeled  internally,  and  so  forth.  Such  detailed  characterizations 
constitute  the  most  explicit  cognitive  models  that  provide  input  to  complex  skills  training 
programs. 

Identification  of  Instructional  Targets.  The  instructional  goal  of  this  early  training  study  was 
to  advance  novice  troubleshooting  capabilities  by  enabling  them  to  use  coordinated  procedural, 
strategic,  and  system  knowledge  in  increasingly  sophisticated  ways.  The  performance 
components  targeted  by  the  instruction  were  those  that  clearly  differentiated  novice  from 
intermediate  and  expert  performances  as  revealed  by  the  PARI  analyses.  They  consisted  of  three 
types  of  procedural  skills,  or  actions  executed  on  the  avionics  test  equipment:  measurements, 
computer  diagnostics,  and  swapping.  Experts  were  able  to  use  all  three  procedures  and  execute 
them  under  the  appropriate  conditions.  Intermediate  technicians  were  less  likely  than  experts  to 
take  measurements  when  appropriate  and  made  heavier  use  of  computer  diagnostics  and 
swapping.  The  preferred  troubleshooting  method  of  novices  was  swapping,  regardless  of  the 
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conditions.  These  performance  differences  provided  a  three-tiered  learning  trajectory  on  which  to 
base  the  instruction. 

The  performance  differences  described  above  can  be  explained  in  terms  of  the  "sufficiency" 
and  the  "efficiency"  of  procedures,  or  actions  taken  on  the  equipment.  A  procedure  is  said  to  be 
sufficient  in  a  particular  situation  if  it  accomplishes  the  current  goal  by  allowing  the  targeted 
circuitry  to  be  thoroughly  examined  and  eliminated  from  further  consideration  as  the  source  of  the 
fault.  A  procedure  is  efficient  if  it  is  the  best  procedure  to  use  among  a  number  of  alternative 
procedures,  each  of  which  is  sufficient  to  accomplish  the  goal.  One  procedure  may  be  more 
efficient  than  another  because  it  takes  less  time,  less  physical  effort,  less  mental  effort,  is  safer, 
provides  more  information  about  the  system,  and  so  on.  In  other  words,  an  efficient  procedure  is 
less  costly  and  more  informative  than  an  inefficient  one  while  producing  the  same  benefit.  Since 
experts  have  more  procedural  options  available  to  choose  from,  they  are  more  concerned  with  the 
efficiency  of  procedures  than  novices  who  tend  to  settle  for  sufficiency  when  choosing 
procedures.  The  sufficiency-efficiency  distinction  makes  salient  the  system  and  strategic 
knowledge  associated  with  the  use  of  each  procedure  because  it  is  this  knowledge  that  gives  rise 
to  the  availability  of  procedural  options. 

To  illustrate,  consider  the  system  knowledge  demands  imposed  by  swapping,  the  use  of 
computer  diagnostics,  and  measurement  taking.  Given  that  the  goal  of  all  troubleshooting  actions 
is  to  investigate  some  part  of  a  signal  path,  then  all  three  procedures  require  at  a  minimum  the 
ability  to  identify  the  components  on  the  circuit  path  to  be  investigated.  Swapping  requires  the 
least  amount  of  related  knowledge  for  its  execution.  It  is  the  preferred  method  of  novices 
presumably  because  of  the  low  knowledge  demands.  One  must  simply  identify  active  components 
and  serially  swap  them  until  the  fault  is  eliminated.  By  comparison,  using  computer  diagnostics  to 
eliminate  parts  of  the  circuitry  from  consideration  requires  more  system  knowledge  since  the 
technician  must  know  something  about  the  system  functionality  in  order  to  select  an  appropriate 
diagnostic  test  to  run.  Further,  to  interpret  its  result,  one  must  know  how  the  circuit  used  in  the 
diagnostic  software  compares  to  the  circuit  in  which  the  fail  originally  appeared. 
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Measurement  taking  requires  still  more  system  knowledge  since  the  technician  must  be  able 
to  manually  manipulate  the  equipment  to  access  the  circuitry  to  measure.  This  means  determining 
details  of  the  faulty  circuit,  such  as  pin  numbers  on  cables  or  circuit  cards  or  relay  contact 
numbers.  In  addition,  taking  measurements  requires  knowledge  of  the  signal  properties  one 
should  test  for,  and  selection  of  an  appropriate  measurement  device.  There  is  also  a  key 
companion  procedural  skill  associated  with  measurement  taking,  namely  the  capability  to  access 
these  system  details  in  the  technical  documentation,  a  skill  that  is  not  necessarily  required  to  run 
computer  diagnostics. 

In  sum,  choosing  an  efficient  procedure  depends  on  knowledge  of  the  costs  associated  with 
using  alternative  procedures,  which  in  turn  depends  on  knowledge  of  the  system.  Although 
swapping  may  be  a  sufficient  procedure  to  use  under  a  wide  variety  of  circumstances,  it  is  not 
always  efficient  since  (for  example)  it  may  cause  damage  to  equipment,  it  may  be  physically 
difficult,  or  the  cost  to  replace  the  swapped  component  may  be  high.  If  an  appropriate  diagnostic 
is  available  under  those  conditions,  using  the  computer  to  test  the  targeted  circuitry  might  be 
more  efficient.  If  not,  then  efficiency  may  dictate  taking  a  measurement. 

Based  on  these  PARI  findings,  three  performance  levels  were  identified  as  the 
approximations  of  skilled  performance  to  be  tutored  in  this  proof-of-principle  study.  At  the  first 
level,  the  instructional  goal  was  to  teach  trainees  how  to  identify  the  components  on  the  signal 
path  where  a  fail  had  been  encountered.  At  this  initial  level  the  task  was  to  build  a  device 
representation  of  the  failed  test  as  a  mental  model  for  guiding  later  troubleshooting  actions.  This 
instructional  focus  is  consistent  with  our  findings  that  experts'  system  knowledge  provides  the 
basis  for  their  procedural  flexibility.  At  the  second  level,  the  instructional  goal  was  to  get  students 
to  choose  a  procedure  for  investigating  each  component  on  the  active  path  that  was  sufficient  for 
eliminating  it  as  the  source  of  the  fault.  At  the  final  level,  the  goal  was  to  teach  trainees  which 
procedure  was  most  efficient  for  investigating  each  component  under  the  prevailing 
circumstances. 
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Training  Methodology.  The  study  was  conducted  as  follows.  A  human  tutor  posed 
troubleshooting  problems  to  novice  technicians  who  were  then  asked  to  isolate  the  fault  step  by 
step.  PARI  records  were  generated  by  the  tutor  who  asked  technicians  at  each  step  for  an  Action, 
a  reason  for  the  Action  (Precursor),  and  what  the  outcome  (Result)  of  the  Action  meant 
(Interpretation).  Trainees  were  instructed  to  generate  device  model  sketches  to  illustrate  their 
solution  steps.  Each  technician  received  individual  tutoring  on  three  to  five  troubleshooting 
problems,  each  requiring  approximately  one  hour  to  solve.  The  number  of  problems  presented 
depended  on  how  many  problems  it  took  for  a  given  subject  to  move  through  the  three 
instructional  levels  described  earlier. 

Each  of  the  instructional  goals  was  pursued  individually  in  the  tutoring  sessions.  A  trainee 
was  required  to  demonstrate  if  s/he  could  reliably  meet  the  first  goal  (identify  the  components  on 
the  active  circuit  path)  before  tutoring  on  the  second  goal  (sufficiency  of  procedures)  was 
addressed,  and  so  on.  The  human  tutor  determined  when  an  instructional  goal  was  met  by  using 
rules  to  diagnose  students'  weaknesses  at  each  level.  For  example,  a  trainee  was  judged  as 
knowing  how  to  identify  components  on  the  active  path  if  her/his  problem  solution  contained 
explicit  references  to  the  components  that  satisfied  the  major  functionalities  of  the  system,  namely, 
signal  generation,  routing,  and  measurement.  Similar  diagnostic  rules  were  developed  for  each  of 
the  other  two  instructional  levels  as  well. 

Each  diagnostic  rule  had  associated  with  it  a  set  of  "querying"  hints  that  provided  a 
structure  for  the  tutor  to  engage  in  a  type  of  Socratic  dialogue  with  the  student.  This  occurred 
when  a  trainee  needed  assistance  in  meeting  a  given  instructional  goal  (i.e.,  when  a  weakness  was 
diagnosed).  The  content  of  these  hints  was  derived  from  the  skill  definitions  for  the  three 
troubleshooting  procedures.  The  structure  and  form  of  the  hints  were  driven  by  pedagogical 
principles  such  as  scaffolding,  active  learning,  and  teaching  of  global  before  local  skills.  The  first 
hint  simply  queried  the  student  for  the  knowledge  lacking  in  her/his  problem  solution,  e.g.,  what  is 
the  measurement  device  in  this  problem?  If  s/he  failed  to  answer  the  question  correctly,  a  second 
query  hinted  at  where  the  necessary  information  could  be  found.  If  the  student  still  failed  to 
generate  the  necessary  information,  s/he  was  directly  told  where  the  information  could  be  found. 
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If  this  query  was  unsuccessful  in  eliciting  the  information,  the  student  was  told  what  the  missing 
information  was,  and  then  asked  why  an  expert  would  find  that  information  relevant  in  solving  this 
problem.  If  unable  to  give  the  reason,  the  student  was  told  the  reason  by  the  tutor.  An  example 
of  the  type  of  tutor-trainee  interchange  that  occurred  is  shown  in  Table  8. 


Table  8 

Sample  student-tutor  dialogue  (Pokomy,  in  preparation). 


This  interchange  begins  when  the  student  is  diagnosed  as  having  a  weakness  in  his  ability  to 
identify  the  components  of  the  active  circuit  path  (Instructional  Goal  1).  The  student  incorrectly 
interprets  a  piece  of  technical  data,  in  this  case,  a  computer  statement,  as  specifying  the  path  of 
the  stimulus  signal  rather  than  the  path  of  the  measurement  signal.  The  tutor  directs  the  student’s 
attention  to  where  the  needed  information  can  be  found.  The  computer  listing  is  shown  below  for 
clarification,  along  with  a  diagram  of  the  circuit  involved  in  this  fault. 


TUTOR:  Do  you  think  the  signal  is  going  into  TPA1 1  [a  relay]  from  the  UUT  or  going 

out  of  it  toward  the  UUT? 


STUDENT :  I  think  it's  going  out. 

TUTOR:  O.K.  Look  at  this  ROUTE  statement  again  where  it  talks  about  TPA1 1 .  What 

is  it  hooking  up?  [The  tutor  is  attempting  to  get  the  student  to  determine 
whether  the  path  through  TPA1 1  is  the  stimulus  or  the  measurement  path.] 

STUDENT:  It's  routing  the  signal  through  TPA1  to  FREQ  1,  if  I’m  reading  it  right. 

TUTOR:  How  is  it  that  you  have  identified  two  paths  through  this  one  relay  [one  path 

going  to  the  UUT  and  one  path  to  FREQ  1]? 

STUDENT:  For  the  waveform  generator  [a  stimulus  device],  you  have  a  CONNect 

statement,  so  there  is  a  relay  that  is  connected— the  stimulus  device  would 
usually  be  sending  something  out  to  the  UUT. 

TUTOR:  So  you've  got  the  ROUTE  statement,  and  this  CONNect  statement  doing 

something.  What  is  the  ROUTE  statement  doing? 

STUDENT:  The  ROUTE  statement  is  routing  the  signal  through  TPA1 1,  so  I  guess  it 
would  be  input  to  the  measurement  device. 

PAPA  LISTING 


SWITCHING  COMPLEX 

DISCONN  •WFG'  OUTPUT  NO  SP 
PROG  ■WFG’  FOR  3.3  V,  1  HZ  FREQ, 

SINE  WAVE,  NO  OFFSET  NORMAL 
CONTIN  MODE,  INTERNAL 
ROUTE  'FREQ1'  INPUTB  FROM  TPA 
DIRECT  FEED  IN  (11)  TP 
CONN  TAIFG’  OUTPUT  NO  SP 
START  'FREQ1' 

WAIT  FOR  2  SEC 
READ  'FREQ1' 

CMP  RESULT,  LL  40  MSEC 
IF  (GO),  GOTO  156500 
DISPLAY  ’UUT  FAILED  TEST  1560’ 
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To  evaluate  the  effectiveness  of  this  cognitive  model-based  instruction,  seven  novice  Air 
Force  avionics  technicians  were  administered  verbal  troubleshooting  tests  in  an  immediate 
posttest,  as  well  as  a  delayed  posttest.  The  PARI  procedures  were  again  used  in  developing  the 
test  problems  and  in  collecting  the  data.  The  troubleshooting  test  problems  were  different  on 
each  occasion,  but  belonged  to  the  same  class  and  difficulty  of  problems  on  which  the  trainees  had 
been  tutored.  Progress  of  their  learning  was  assessed  both  in  terms  of  the  sufficiency  of  their 
action--that  is,  whether  they  sufficiently  investigated  all  suspect  pieces  of  the  equipment  (levels  1 
and  2  of  the  instruction)~as  well  as  the  efficiency  of  their  actions— that  is,  whether  they  efficiently 
collected  relevant  information  while  conserving  time  and  resources. 

Results.  Results  showed  statistically  significant  improvements  in  both  areas,  with 
particularly  dramatic  gains  in  efficiency.  Mean  scores  are  plotted  in  Figure  8.  The  group's 
sufficiency  in  examining  all  suspect  parts  of  the  equipment  improved  from  a  pretest  mean  of  84 
percent  correct  to  a  posttest  mean  of  100  percent.  The  delayed  posttest  mean  was  also  100, 
indicating  the  improvement  was  retained  over  several  days  (a  weekend).  The  group's  efficiency  in 
fault  isolation  improved  over  twofold.  The  pretest  mean  for  efficiency  was  37;  the  initial  posttest 
mean  was  92,  and  the  delayed  posttest  mean  was  93. 


Pre  Post  Delayed  Post 

TROUBLESHOOTING  TEST 

Figure  8.  Mean  sufficiency  and  efficiency  scores  on  pre-  and  posttests. 
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Although  no  control  group  was  used  in  this  study,  these  results  are  consistent  with  the 
conclusion  that  data  from  a  PARI  analysis  can  effectively  focus  instruction  on  the  unobservable 
decision  processes  that  account  for  efficient  solutions  and  the  reasons  behind  problem  solving 
steps.  The  course  of  skill  acquisition  is  likewise  made  salient  in  cognitive  models  across 
proficiency  levels,  which  appeared  in  this  study  to  enable  more  informed  decisions  about 
instructional  sequencing.  The  study  described  in  the  following  discussion  provides  perhaps  a 
stronger  test  of  the  PARI  methodology  as  an  aid  in  instructional  design. 

Development  and  Evaluation  of  an  Avionics  Troubleshooting  Tutor 

Instructional  Design.  A  second  example  of  a  PARI-based  training  study  is  provided  by  the 
successful  development  and  field  test  of  an  intelligent  tutoring  system  designed  to  accelerate  the 
acquisition  of  troubleshooting  skill  in  one  of  several  Air  Force  avionics  jobs.  (See  Gott,  1989  and 
Lajoie  &  Lesgold,  1990  for  a  detailed  account.)  The  curriculum  goals  and  content  of  the  tutor 
were  derived  from  detailed  PARI-like  cognitive  analysis  that  contrasted  the  performance  of  skilled 
with  less-skilled  apprentice  technicians  on  realistic  fault  isolation  tasks  (Gitomer,  1984,  1988; 
Glaser  et  al,  1985;  Gott,  Bennett,  &  Gillet,  1986).  Performances  for  this  range  of  technicians 
were  represented  with  a  problem  space-based  formalism  (Newell  &  Simon,  1972),  in  which 
alternative  sequences  of  PARI  steps  were  represented  in  a  hierarchical  structure  to  provide 
multiple  solution  traces  through  a  problem  space  (Glaser  et  al,  1985).  From  multiple  traces,  it 
was  possible  (for  a  given  classes  of  problems)  to  abstract  the  hierarchies  of  plans  and  actions  used 
by  most  technicians.  The  intelligent  troubleshooting  tutor  (called  Sherlock)  was  built  from  these 
prototypical  (PARI)  structures.  Its  design  is  based  on  the  same  approach  underlying  the  proof-of- 
principle  study:  expert  models  of  performance  provided  the  ideals  used  as  instructional  goals; 
less-than-expert  performance  data  provided  the  basis  for  a  curriculum  progression;  finally,  the 
instruction  took  place  in  an  active,  problem-oriented  learning  environment  designed  to  foster 
complex  skill  acquisition. 

Instructional  Targets.  Sherlock's  principle  instructional  goal  is  tied  to  the  cognitive  activity 
that  is  required  to  understand  the  processes  that  define  a  test  being  run  on  the  avionics  test 
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equipment.  These  processes  were  described  in  the  Stage  V  PARI  procedures  (see  Section  IE, 
Figure  6).  Their  understanding  requires  that  the  technicians  identify  the  active  components  of  the 
circuit  involved  in  the  test.  The  reader  may  recall  that  identification  of  active  equipment 
components  was  also  the  first  instructional  goal  tutored  in  the  proof-of-principle  study.  Lesgold, 
Lajoie,  Bunzo,  and  Eggan  (1988)  refer  to  this  instructional  goal  as  the  fundamental  mental  model 
to  be  tutored— the  mental  model  of  an  electronic  test.  Sherlock's  theory  of  problem  solving 
follows  from  this  focus:  troubleshooting  is  viewed  as  "device  model-guided  plans  and  actions  that 
are  regulated  by  executive  (strategic)  control  processes."  The  notion  of  an  electronic  test  is  the 
conceptual  base  for  the  procedural  and  strategic  knowledge. 

In  addition  to  targeting  the  fundamental  mental  model  of  the  test,  the  tutor  is  also  directed 
toward  developing  the  technician's  goal  structure  (plans)  for  investigating  the  equipment  (given 
the  testing  process),  the  procedural  knowledge  in  the  form  of  specific  fault  isolation  actions  that 
instantiate  the  top-level  goal  structure,  and  additional  strategic  (control)  knowledge  to  inform 
decision  making  and  regulate  systematicity  during  problem  solving.  These  instructional  goals 
were  identified  as  pervasive  troubleshooting  weaknesses  among  apprentices  in  the  prior 
comparative  cognitive  task  analyses  (see  Gitomer,  1984,  1988;  Glaser  et  al.,  1985).  Those 
weaknesses  were  also  identified  in  the  cognitive  modelling  of  less  skilled  performers  in  the  proof- 
of-principle  study  and  served  there  as  the  basis  for  instructional  targets  as  well. 

Tutor  Evaluation.  Sherlock  was  evaluated  in  a  controlled  experiment  at  two  geographically 
separated  Air  Force  F-15  flying  wings.  A  total  of  32  trainees  were  tested  on  a  number  of 
technical  proficiency  indicators  to  establish  matched  experimental  and  control  groups  at  each  site. 
The  principal  form  of  assessment  used  in  both  pre-and  posttesting  was  the  Verbal 
Troubleshooting  Tests.  A  paper-and-pencil  noninteractive  test  comprised  of  mini  (focused) 
troubleshooting  scenarios  was  also  used.  In  addition,  technicians  completed  a  post-tutor 
evaluation  questionnaire. 

During  the  intervention,  the  experimental  group  spent  an  average  of  20  hours  working 
Sherlock's  34  problems.  Tutoring  sessions  were  scheduled  in  2-  to  3-hour  blocks  that  spanned  an 
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average  of  12  working  days.  The  control  group  continued  participating  in  the  existing  (informal) 
on-the-job  training  program  during  that  period. 

Tutor  Effectiveness.  Both  combined  and  site-specific  data  analyses  were  conducted 
(Nichols,  Pokomy,  Jones,  Gott,  &  Alley,  in  press).  Pretest  and  posttest  means  and  standard 
deviations  for  control  (N=16),  experimental  (N=16)  and  an  advanced  group  of  experienced 
technicians  (N=13)  are  reported  in  Table  9.  Independent  sample  t-tests  showed  no  significant 
difference  between  the  experimental  and  control  groups  on  the  pretest,  while  regression  analyses 
using  posttest  scores  indicate  a  highly  significant  effect  due  to  the  tutor.  As  Table  9  shows,  the 
difference  between  posttest  means  was  over  20  points,  with  experimental  subjects  moving  to 
within  three  points  of  the  advanced  group.  The  comparative  experience  (in  months)  for  the  three 
groups  was  28  months  (experimental  airmen),  37  months  (control  airmen),  and  114  months 
(advanced  airmen).  In  site-specific  and  other  related  analyses  (e.g.,  paper-and-pencil  test),  the 
margin  of  difference  between  experimental  and  control  groups  was  comparable  (ranging  form  14 
to  26  points). 


Table  9 

Pre-  and  posttest  means  and  standard  deviations  from  avionics  troubleshooting  tutor  evaluation  (Gott, 
1989). 


Group 

Pretest 

Posttest 

Mean 

SD 

Mean 

SD 

Experimental  Group 

56.93 

28.72 

79.00 

17.39 

(Tutor  Group) 

Control  Group 

53.40 

22.38 

58.88 

19.67 

(On-the-Job  Training  Group) 

Advanced  Group 

82.15 

12.27 

(Skilled  Airmen) 
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These  results  support  the  claim  that  Sherlock  was  effective  in  teaching  avionics 
troubleshooting  and  accelerating  the  acquisition  of  troubleshooting  experience.  In  an  attempt  to 
gain  a  better  understanding  of  what  might  have  accounted  for  the  tutor's  effectiveness,  technicians 
who  participated  in  the  study  were  asked  to  give  their  reactions  to  some  of  Sherlock's  features  in 
a  post-tutor  evaluation  questionnaire.  In  this  evaluation,  Sherlock  received  its  highest  marks  for 
its  effectiveness  in  teaching  an  approach  to  troubleshooting,  and  in  improving  technicians' 
understanding  of  the  (stimulus  and  measurement)  functional  areas  of  the  equipment  and  of  the 
routing  of  circuitry  in  general.  The  tutor  received  comparatively  lower  marks  for  freedom  of 
troubleshooting  moves  within  the  tutor,  usefulness  and  timeliness  of  hints  and  assistance,  and 
meaningfulness  of  feedback  at  the  end  of  the  session. 

The  instructional  focus  on  understanding  the  processes  involved  in  testing  a  piece  of 
equipment  (i.e.,  the  fundamental  mental  model  of  a  test)  thus  appeared  to  be  responsible  for  the 
training  effects  observed  in  this  study  and  the  training  study  described  earlier.  By  targeting 
system  understanding  in  trainees,  this  focus  provided  the  conceptual  base  for  the  acquisition  of 
procedural  and  strategic  knowledge.  In  both  studies,  the  PARI  methodology  was  instrumental  in 
identifying  the  instructional  targets  and  assessing  the  extent  to  which  they  fostered  a  deep 
understanding  of  equipment  functioning.  Next,  we  describe  a  study  that  addresses  how  PARI- 
based  cognitive  models  provide  input  to  training  designed  to  foster  breadth  of  system 
understanding. 

Transfer  of  Technical  Knowledge 

Rationale.  The  rapidly  advancing  technologies  that  populate  modem  work  environments 
demand  considerable  mental  adaptiveness  from  human  operators  and  maintainers.  Workers  in 
technical  domains  such  as  electronics  must  typically  master  a  broad  array  of  complex  systems  as 
well  as  adapt  to  a  continuous  stream  of  the  latest  releases/generations/models  of  those  systems. 

In  psychological  terms,  the  performer  must  be  a  good  transferer  of  knowledge  and  skill  to  be 
effective.  In  the  context  of  the  BJS  Program,  developing  training  that  fosters  mental  adaptiveness 
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is  particularly  critical  since  current  Air  Force  restructuring  policies  have  resulted  in  broadened 
maintenance  responsibilities  that  require  technicians  to  maintain  multiple  aircraft  subsystems. 

Given  the  assumption  that  the  ability  to  solve  novel  troubleshooting  problems  on  one 
equipment  system  requires  a  deep  understanding  of  that  system,  a  reasonable  extension  of  that 
claim  is  that  the  same  type  of  understanding  provides  an  advantage  when  learning  to  troubleshoot 
a  novel  equipment  system.  A  recent  training  study  therefore  addressed  the  issue  of  how 
technicians'  understanding  of  one  equipment  system  influenced  their  learning  of  a  related,  but 
relatively  unfamiliar  system,  and  their  ability  to  troubleshoot  the  unfamiliar  equipment  (Gott,  Hall, 
Pokomy,  Dibble,  &  Glaser,  1990).  Technicians  participating  in  the  study  had  from  three  to  ten 
years  of  experience  in  one  of  two  avionic  specialties  (their  "home"  job)  and  at  the  time  of  the 
study,  were  crosstraining  into  a  third  avionic  specialty  (the  "target,"  or  new  job).  The  study  thus 
provided  an  opportunity  to  examine  learning  and  transfer  in  a  real-world  task  for  which  learners 
had  a  great  deal  of  relevant  prior  knowledge.  The  PARI-based  cognitive  models  of  the  three 
avionics  jobs  served  as  a  framework  within  which  to  interpret  individual  differences  in  learning 
behavior  and  subsequent  troubleshooting  performance,  as  well  as  a  basis  for  predicting  which 
aspects  of  the  new  job  would  pose  the  greatest  impediments  to  performance. 

The  avionics  jobs  that  this  study  focussed  on  differ  primarily  with  respect  to  the  test  stations 
used  in  each  job.  Although  all  stations  serve  the  same  general  function  (testing  LRU's,  or  black 
box  components,  from  the  jets),  each  works  on  different  sets  of  LRUs.  As  a  result,  each  avionics 
job  has  unique  types  of  test  stations.  The  functions  of  the  test  stations  are  to  simulate  the 
electronic  signals  that  the  LRU  would  receive  if  it  were  in  the  airplane,  and  to  measure  the  signal 
it  produces.  The  test  station  thus  tests  the  LRU  by  sending  every  signal  it  is  capable  of  receiving 
and  then  determining  whether  it  is  responding  correctly.  When  a  test  fails,  the  technician  must 
identify  the  source  of  the  problem  and  repair  or  replace  the  faulty  element. 

When  the  fault  lies  in  the  LRU,  problem  solving  (i.e.,  troubleshooting)  is  relatively  simple, 
because  any  given  LRU  has  a  limited  range  of  functions  and  components.  When  the  fault  lies  in 
the  test  package  (the  cables  and  apparatus  which  serve  as  an  interface  between  the  LRU  and  the 
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test  station),  troubleshooting  can  also  be  achieved  by  routinized  procedures.  However,  when  the 
fault  lies  in  the  test  station  troubleshooting  is  complicated  by  the  size  of  the  station— 
approximately  98  -  324  cubic  feet  of  electronic  components;  by  the  tremendous  array  of  signal 
generation,  signal  measurement,  and  signal  routing  functions  it  serves;  and  by  the  fact  that  very 
little  of  what  the  test  station  does  is  visible  to  the  technician.  The  cognitive  models  generated  for 
the  three  avionics  jobs  now  being  consolidated  under  Air  Force  restructuring  policies  describe  the 
problem-solving  demands  associated  with  troubleshooting  these  test  stations. 

Comparison  of  PARI-based  Cognitive  Models.  Figures  9  through  11  show  skeletal 
cognitive  models  for  each  of  the  three  avionics  jobs  ("manuals",  autos",  and  "EWS",  respectively) 
at  an  intermediate  level  of  representation.  In  all  three  jobs,  system,  procedural,  and  strategic 
knowledge  must  be  coordinated  for  troubleshooting  to  be  effective.  For  each  type  of  knowledge, 
the  expert  has  access  to  elaborate  hierarchical  knowledge  structures  that  range  from  specific 
knowledge  instantiations  to  abstractly  stated  principles. 

A  comparative  analysis  of  these  models  reveals  the  dissimilarities  that  exist  at  this 
intermediate  level  of  specificity.  Dashed  lines  on  the  figures  denote  the  knowledge  components 
that  are  demanded  by  the  computerized  test  stations  (automatic  and  EWS  stations)  but  not  by  the 
manually  operated  station  (the  manual  test  station).  At  lower  levels  of  specificity,  similarities 
between  the  two  computerized  jobs  would  also  disappear.  On  the  basis  of  such  comparisons,  the 
EWS  job  was  chosen  as  the  target  job  in  this  study  because  the  differences  between  EWS 
troubleshooting  demands  and  the  other  two  jobs  are  greater  than  for  any  other  pairings  of  the 
three  jobs.  In  particular,  considerable  differences  show  up  at  lower  levels  of  specificity  between 
the  EWS  demands  vs  the  other  two  models  in  these  areas:  the  System  Knowledge  subcomponent 
"Equipment  Structure"  and  the  Procedural  Knowledge  subcomponent  "Actions  Using  Tech 
Data."  The  models  thus  predict  that  these  two  aspects  of  the  EWS  job  pose  impediments  to 
troubleshooting  performance  for  technicians  crosstraining  from  manuals  or  autos. 
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Figure  9.  Cognitive  model  for  manual  avionics  equipment  troubleshooting. 


Method.  Three  manuals  and  three  autos  technicians  were  chosen  for  participation  in  the 
study  on  the  basis  of  a  pretest  in  their  home  jobs  and  the  amount  of  experience  they  had  with  the 
EWS  test  equipment.  All  subjects  had  only  a  standard  lecture  course  on  the  EWS  equipment  and 
very  little  (one  to  two  months)  hands-on  experience.  In  addition  to  being  EWS  crosstrainees, 
technicians  participating  in  the  study  were  required  to  pass  a  verbal  troubleshooting  pretest  in 
their  home  jobs  by  solving  all  of  the  troubleshooting  problems  presented.  No  criteria  for  the 
quality  of  those  solutions  were  used  as  a  basis  for  technicians'  participation,  only  their  ability  to 
independently  solve  the  home  job  problems  without  the  aid  of  an  expert.  This  ensured  that 
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Figure  10.  Cognitive  model  for  automatic  avionics  equipment  troubleshooting. 

subjects  had  enough  knowledge  of  their  home  jobs  to  transfer  to  the  new  job  that  they  would  not 
be  overwhelmed  by  the  novelty  of  EWS  troubleshooting  task.  Thus,  while  technicians  were  fairly 
equal  with  respect  to  their  experience  in  the  EWS  job,  they  did  not  necessarily  have  the  same 
degree  or  type  of  knowledge  of  their  home  jobs. 

The  study  consisted  of  pre-  and  post-tests  on  EWS  troubleshooting  problems,  separated  by 
a  learning  phase  in  which  subjects  were  allowed  to  ask  questions  of  an  EWS  expert  as  they  solved 
learning  problems.  Subjects  were  instructed  to  ask  whatever  questions  they  needed  answered  in 
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Figure  11.  Cognitive  model  for  EWS  equipment  troubleshooting. 

order  to  make  progress  toward,  and  ultimately  achieve  a  solution  to  the  learning  problems.  They 
were  also  told  that  they  would  be  asked  later  to  solve  a  set  of  posttest  problems  without  the  help 
of  the  expert.  The  posttest  problems  varied  in  similarity  to  problems  presented  in  earlier  phases  of 
the  study.  Subjects  were  therefore  encouraged  to  also  ask  questions  during  the  learning  phase 
that  would  be  generally  useful  in  solving  a  wide  variety  of  EWS  problems,  and  not  restrict  their 
questions  to  those  required  to  solve  the  learning  problems  specifically.  All  verbal  troubleshooting 
data  were  collected  using  the  standard  PARI  format.  Subjects'  questions  to  the  expert  were  also 
recorded,  as  were  the  experts'  answers. 
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Results.  In  order  to  determine  which  aspects  of  the  EWS  job  posed  the  greatest 
impediments  to  troubleshooting  performance,  an  analysis  was  performed  on  the  types  of  questions 
asked  most  frequently.  This  analysis  involved  the  classification  of  subjects'  question  in  terms  of 
the  categories  represented  in  the  EWS  model  depicted  in  Figure  11.  The  results  showed  that  the 
great  majority  of  subjects'  questions  had  to  do  with  those  aspects  of  the  EWS  job  shown  by  the 
models  to  be  most  different  from  the  manuals  and  autos  jobs:  the  System  Knowledge 
subcomponent  "Equipment  Structure"  and  the  Procedural  Knowledge  subcomponent  "Actions 
Using  Tech  Data." 

However,  subjects  differed  with  respect  to  their  degree  of  improvement  in  troubleshooting 
the  EWS  equipment  as  measured  by  pre-  and  posttest  difference  scores.  The  mean  pre  and 
posttest  difference  scores  for  the  four  subjects  classified  as  better  learners  (those  who  showed  the 
greatest  pre-  to  posttest  improvement),  and  the  two  subjects  classified  as  poorer  learners,  are 
shown  in  Figure  12.  The  better  learners  also  appeared  to  be  better  transferers  in  the  sense  that 
they  were  able  to  solve  a  wider  variety  of  troubleshooting  problems  in  the  posttest.  Given  the 
quantitative  differences  found  in  subjects'  EWS  troubleshooting  performance,  associated 
qualitative  differences  were  examined.  These  analyses  focussed  on  the  learning  behavior  exhibited 
during  the  learning  phase,  and  the  ways  in  which  prior  knowledge  from  subjects'  home  jobs  was 
applied  in  solving  the  EWS  problems. 

All  subjects  clearly  brought  relevant  prior  knowledge  to  the  EWS  troubleshooting  task.  In 
general,  this  knowledge  took  the  form  of  mental  models  of  the  test  equipment  and  of  the 
troubleshooting  task  itself.  These  models  were  revealed  in  subjects'  PARI  solutions  to  troubleshooting 
problems  in  both  their  home  jobs  and  in  the  EWS  job  in  the  manner  described  in  Section  m  of  this 
paper.  For  the  better  learners,  these  models  were  abstract  representations  that  were  used  as 
interpretive  structures  by  good  learners  to  interrogate  the  expert.  These  subjects  constantly  evaluated 
the  adequacy  of  their  models  by  monitoring  their  own  comprehension  of  the  problems  and  asked 
questions  to  elaborate  and  adapt  their  models  to  the  new  domain.  Thus,  a  larger  proportion  of  these 
subjects'  questions  had  to  do  with  aspects  of  equipment  structure  and  function  than  those  of  poorer 
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Figure  12.  Mean  pre-  and  posttest  scores  for  better  and  poorer  learners. 

learners  whose  questions  focussed  primarily  on  procedures  for  accessing  technical  data.  The 
learning  strategy  of  acquiring  deeper  system  knowledge  clearly  distinguished  the  better  and 
poorer  transferers,  suggesting  that  system  understanding  provided  the  flexibility  required  to  solve 
the  posttest  transfer  problems. 

Poorer  learners'  prior  knowledge  appeared  to  be  represented  in  a  job-specific  (rather  than 
abstract)  form.  As  a  result,  they  failed  to  adapt  effectively  to  the  new  domain  as  seen  in  their 
tendency  to  overgeneralize  concepts  from  their  home  jobs.  Because  these  subjects  failed  to  think 
in  terms  of  how  differences  in  the  EWS  equipment  might  influence  a  good  technicians' 
troubleshooting  strategy,  they  demonstrated  a  high  degree  of  procedural  rigidness  when  solving 
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problems.  These  technicians  employed  the  same  strategies  they  tended  to  use  in  their  home  jobs, 
and  refused  to  deviate  from  them  even  when  prompted  to  do  so  by  the  experts.  This  is  consistent 
with  the  fact  that  their  questions  focussed  almost  exclusively  on  how  to  access  the  technical  data 
they  needed  to  execute  troubleshooting  procedures,  rather  than  on  understanding  how  the  EWS 
test  equipment  works  and  how  it  differs  from  that  in  their  home  jobs.  These  preliminary  data 
support  the  notion  that  in  the  domain  of  troubleshooting,  the  basis  of  procedural  flexibility  and 
adaptability  to  novel  situations  lies  in  a  deep  understanding  of  the  system  that  can  accommodate 
both  abstract  representations  that  can  be  generalized  across  equipment  systems,  as  well  as  highly 
elaborated  models  tailored  to  the  specifics  of  particular  problem-solving  situations. 

Conclusions.  These  results  are  compatible  with  Brown's  conclusion  that  "wide  patterns  of 
generalization,  flexible  transfer,  and  creative  inferential  projections  are  all  indices  of  deeper 
understanding  of  causal  mechanism"  (1990;  p.  129):  technicians  who  directed  their  learning 
toward  a  deeper  understanding  of  the  EWS  system,  and  thus,  the  causal  mechanisms  underlying 
the  problem-solving  task,  showed  the  greatest  improvement  in  EWS  troubleshooting  and  solved  a 
wider  variety  of  EWS  problems.  The  depth  and  quality  of  technicians'  knowledge  in  their  home 
jobs  appeared  to  exert  a  strong  influence  on  the  type  of  EWS  knowledge  technicians  thought  they 
would  find  useful,  and  thus  on  the  degree  of  transfer. 

This  study  contrasts  with  the  large  majority  of  laboratory  studies  reported  in  the  transfer 
literature  in  that  it  represents  what  Gott  et  al.  refer  to  as  a  "naturalistic  transfer"  study  (1990;  in 
preparation).  That  is,  all  subjects  had  relevant  knowledge  to  bring  to  the  learning  situation  and 
whether  or  not  they  would  apply  it  was  not  an  issue  in  this  study.  The  question  was  how  the 
depth  and  quality  of  that  knowledge  influenced  learning  behavior  and  subsequent  transfer. 

Because  of  the  complexity  of  knowledge  associated  with  typical  real-world  tasks,  many 
laboratory  studies  of  learning  and  transfer  use  tasks  that  are  trivial  in  terms  of  the  knowledge 
demands  they  impose  (e.g.,  crossing  out  p's,  then  q's  in  text,  or  sorting  shapes,  then  colors).  Such 
studies  have  been  criticized  because  they  have  little  or  no  functional  value  to  subjects  making  it 
unclear  why  one  would  expect  retention  and  transfer  of  such  learning.  One  could  argue,  however, 
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that  inconsequential  tasks  are  used  in  laboratory  studies  precisely  because  they  are  trivial:  since 
they  require  so  little  knowledge,  the  researcher  has  a  better  chance  of  adequately  characterizing 
what  is  to  be  transferred  from  the  learning  task,  and  controlling  the  task-relevant  knowledge  that 
subjects  have  available  to  transfer.  Thus,  tasks  are  chosen  for  which  subjects  can  be  given  all 
relevant  knowledge  in  a  "learning  phase"  and  for  which  subjects'  prior  knowledge  is  irrelevant. 
Such  tasks  are,  by  definition,  trivial  and  inconsequential. 

The  problem  with  those  studies  is  that  they  fail  to  acknowledge  that  real-world  learning 
does  not  take  place  in  a  cognitive  vacuum  where  no  prior  knowledge  is  brought  to  bear.  Their 
results  have  limited  value  if  they  do  not  generalize  to  the  kinds  of  learning  tasks  that  people 
confront  in  their  everyday  lives.  The  naturalistic  study  of  transfer  described  above  demonstrates 
how  the  ability  to  characterize  the  content  of  the  knowledge  that  individuals  bring  to  cognitively 
complex,  and  moreover,  meaningful  tasks  allows  one  to  draw  substantive  conclusions  about  what 
is  transferred,  and  how  the  quality  and  structure  of  the  knowledge  base  influence  the  transfer 
process.  The  important  thing  to  note  in  the  current  context  is  that  the  content  of  subjects' 
knowledge  was  identified  using  the  PARI  methodology,  and  interpreted  within  the  framework  of 
PARI-based  cognitive  models  of  avionics  experts.  The  tools  provided  by  the  task  analysis 
procedures  thus  allow  the  study  of  learning  and  transfer  in  tasks  involving  highly  developed, 
domain-specific  knowledge. 


Summary 

The  studies  described  in  this  section  constitute  tests  of  the  PARI-based  cognitive  models 
and  thus  of  the  methodology's  ability  to  adequately  inform  such  models.  The  models  were  used  to 
identify  the  instructional  targets  of  the  proof-of-principle  study  and  the  Avionics  Troubleshooting 
Tutor,  and  to  predict  learning  outcomes  in  the  transfer  study.  The  effectiveness  of  training 
demonstrated  in  the  first  two  studies,  and  the  accuracy  of  the  models'  predictions  found  in  the 
third  study  suggest  that  the  methods  do  in  fact  capture  cognitive  components  of  complex  problem 
solving  tasks,  and  allow  meaningful  conclusions  about  how  those  tasks  should  be  taught.  The 
verbal  troubleshooting  task  used  in  all  three  studies  is  also  based  on  the  PARI  interview  structure. 
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and  provided  models  of  individual  student  performances  which  could  then  be  evaluated  against 
the  expert  model  to  assess  students'  skill.  These  uses  of  the  methodology  reflect  an  accepted 
principle  of  practice-oriented  training,  namely,  that  testing  and  training  mirror  the  criterion 
performance  (Gott,  1987). 

The  methodology  is  being  applied  in  similar  ways  to  the  development  of  a  "Job  Family" 
Tutor  (JFT).  This  prototype,  F-15  avionics  tutor  is  designed  to  teach  troubleshooting  on  the 
three  avionics  test  stations  described  earlier  in  conjunction  with  the  transfer  study.  Like  Sherlock, 
the  JFT  represents  an  example  of  an  intelligent  tutoring  system.  These  systems  provide 
individualized  instruction  by  responding  to  each  student's  particular  strengths  and  weaknesses 
based  on  the  tutor's  diagnosis  of  his/her  skill.  Cognitive  models  of  both  the  expert  and  the  student 
inform  the  instructional  content  of  the  tutor,  and  are  also  used  by  the  tutor  in  student  diagnosis. 

Intelligent  tutoring  systems  represent  a  relatively  recent  advance  in  computer-based  training 
(CBT)  and  differ  from  more  traditional  CBT  systems  in  (among  other  things)  the  way  that 
individualized  instruction  is  achieved:  whereas  traditional  CBT  systems  allow  the  student  to 
control  certain  parameters  of  the  instructional  interaction  (such  as  sequence  and  rate  of 
information  presented),  an  intelligent  tutoring  system  guides  this  interaction  based  on  its  model  of 
the  expert,  its  model  of  the  student,  and  the  individual  student's  actions.  The  instructional 
interaction  in  the  latter  case  more  closely  parallels  that  between  a  student  and  a  human  tutor. 
Although  many  instructional  developers  outside  the  scientific  and  research  community  recognize 
the  advantages  of  this  approach,  they  lack  the  research  tools  to  implement  it.  As  stated  at  the 
outset  of  this  paper,  one  goal  of  the  B  JS  program  is  to  develop  the  tools  needed  for  improved 
instructional  content  and  delivery  and  make  them  available  to  Air  Force  instructional  designers. 
One  step  toward  this  goal  is  the  documentation  of  the  PARI  data  collection  procedures  in  this 
guide. 

The  PARI  methodology  has  now  been  used  by  a  number  of  researchers  and  individuals  in 
the  education  community  to  study  a  variety  of  maintenance  jobs  in  the  Air  Force;  these  include 
maintenance  of  avionic  and  mechanical  subsystems  in  F-15,  F-16,  and  F-l  1 1  aircraft.  A  future 


92 


goal  of  our  Program  is  to  examine  the  applicability  of  this  procedure  to  nonmaintenance  tasks. 
To  the  extent  that  job  environments  in  our  society  are  becoming  increasingly  technical  and 
demanding  in  terms  of  complex  problem  solving,  we  believe  the  procedure  will  have  broad 
generality. 
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Appendix  A 

Training  and  Experience  Questionnaire 

(For  F15  Avionics  Job) 

Name 

Present  Organization 

Rank 

Present  Job 

AFSC 

Months  in  Present  Job 

Phone 

Months  in  AFSC 

1 .  What  are  your  primary  duties  in  your  present  job? 

2.  What  previous  jobs  have  you  had  in  this  AFSC?  Please  describe  briefly. 


3.  What  other  AFSCs  have  you  held?  Please  give  brief  descriptions. 


4.  What  type  of  Air  Force  training  (tech  school,  FTD,  factory  school,  etc.)  have  you  had  in  the 
electrical/electronic  field? 

Type  of  course: _ Length: _ 

Year: _  (Note:  please  continue  on  back  if  necessary.) 

5.  What  type  of  training  outside  the  Air  Force  (for  example,  vocational  ed  program,  associate's  degree 
in  electronics)  have  you  had  in  electrical/electronic  fields? 

Type  of  course: _ Length: _ 
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Year: _  (Note:  please  continue  on  back  if  necessary.) 

6.  In  your  opinion,  what  are  the  most  important  EW  areas  where  experienced  manuals  and  automatics 
personnel  need  training  for  Rivet  Workforce  purposes? 


7.  In  your  opinion,  what  are  the  most  important  EW  areas  where  inexperienced  manuals  and 
automatics  personnel  need  training  for  Rivet  Workforce  purposes? 


8.  What  are  the  primary  EW  training  needs  for  3 -level  personnel  when  they  report  to  your 
workcenter? 
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Appendix  B 

Workplace  Ecology  Questionnaire 


AFS: _ 

SME: _ 

1 .  Effects  of  mobility  exercises 

a.  What  are  the  characteristics  of  exercises  (e.g.,  how  long  are  they,  what  conditions  are 
simulated,  etc.)? 

b.  What  equipment  is  mobilized  and  shipped? 

c.  What  are  the  effects  of  mobility  exercises  on  maintenance  requirements? 


2.  Effects  of  workcenter  equipment  usage 

a.  During  how  many  shifts  would  the  equipment  be  used? 


b.  Is  the  equipment  turned  off  between  shifts?  (Circle  one) 

Yes  No 

3.  Mission  of  organization 

Does  the  workcenter  emphasize  its  operational  or  its  training  mission?  (Check  one) 

Heavy  operational  emphasis 

_ Some  operational  emphasis 

Even  emphasis 

_ Some  training  emphasis 

Heavy  training  emphasis 

4.  Effects  of  organization's  mission 

a.  What  is  the  effect  of  the  organization's  mission  on  the  kinds  of  faults  you  troubleshoot? 


b.  What  is  the  effect  of  the  organization's  mission  on  the  way  you  troubleshoot? 


5.  What  is  the  hit  rate  for  diagnostics? 
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6.  Are  there  temperature-related  effects  on  the  equipment?  (Circle  one) 


Yes  No 


7.  Are  there  faults  that  tend  to  appear  when  the  equipment  is  first  turned  on?  (Circle  one) 

Yes  No 

If  yes,  specify  - 

8.  Are  there  differences  between  bases  with  operational  and  training  missions  in  terms  of  maintenance 
of  this  equipment? 


9.  What  do  first  termers  usually  do  when  the  equipment  fails?  (check  one) 

Call  over  a  5-  or  a  7-level 

_ Consult  AFETS  or  civilian  tech  rep 

Muddle  through 
_ Other  (please  specify) 
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Appendix  C 

Problem  Category  Exemplars  and  Problem  Categories 


Category  Exemplar 

Monitor  Matrix  Relay 

Instrument  Select  Relay 

Interface  Chassis  Relay 

RF  Switch 

Mixer  Diodes 

Pulse  Generator 

I/O  Card 

Coaxial  Switch  Driver 

Time-Delay  Generator 

L-17  Self-Test  Network 


Cause/Effect 


MMX  relay  stuck  open 
causing  PS-3  to  fail  confidence 
test 

Stuck  relay  on  instrument 
select  card  causes 
measurement  bus  failure 

Open  relay  in  I/F  chassis 
results  in  LRU  response  not 
being  routed  to  DMM 

Bad  RF  switch  in  coax 
switching  panel  causes  MS  S3 
failure 

MSS2  failure  due  to  bad  mixer 
diodes 

Bad  pulse  from  pulse 
generator  causes  LRU-3  to  fail 
test  323  but  OA/FI  passes 

Basi  I/O  fails  due  to  bad  I/O 
card 

Coaxial  switching  driver 
malfunctions  causing  pulse 
generator  signal  output  to  be 
routed  incorrectly 

MMX  U20  loads  down  TDG 
real-time  clock  causing  basic 
I/O  to  fail 

DMM  fails  VDC  confidence 
test  but  passes  DC  calibration 
due  to  bad  L-17  self-test 
network 


Problem  Category 

A/D  Switching/Routing 


RF  Switching/Routing 


RF  Waveform  Analysis 
Analog  Waveform  Analysis 


Data  Transmission  (I/O  Logic) 
Digital  Data  Analysis 


Clock/Timing 


Load 
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Category  Exemplar 


Cause/Effect 


Problem  Category 


Unit  4C  Power  Supply 

26  VDC  power  supply  in  unit 
4C  has  low  output  causing  RF 
relays  to  malfunction 

Power  Supply 

PS-9  Zener  Diode 

Bad  zener  diode  for  PS-9 
causes  power  supply 
programming  failure 

Global  error 

Constant  global  error  due  to 
full  disk 

Computer  Error  Messages 

Tape  position 

Tape  header  ID  does  not 
match  keyed-in  data  because 
tape  is  in  wrong  position 

Software  Maintenance 

RF  Cable 

Bad  external  RF  cable  causes 
power  loss  during  testing  in 

RF  drawer 

Connectors 

Probe  Power  Connector 

5  VDC  probe  power  connector 
on  DPO  is  bad  causing  DPO  to 
be  inoperable  in  remote  mode 

Interface  Chassis  Pin  Contacts 

External  test  points  fail  OA/FI 
du  to  dirty  pin  contacts  on  I/F 
chassis 

5VDC  Power  Display 

Keyboard  and  computer  do 
not  have  lighted  displays 
because  power  supply  is 
turned  off 

Initial  Equipment  Setup 

Bootstrap  CCA 

No  “MAKE  DISK  READY” 
indicator  on  CRT  and  JSDTD 
will  not  load  due  to  bootstrap 
hangup 

Computer  Hardware 

Replaced  A28  Card 

LRU-3  fails  test  323  due  to 
replacement  of  A28  card 
which  requires  realignment 

Alignments 
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Appendix  D 


Expert  Solution  Path 


SOLUTION  PATH  ALTERNATIVE  RESULTS/ 

INTERPRETATIONS 
(REHASH  #2) 


ALTERNATIVE 
ACTIONS 
(REHASH  #3) 


ALTERNATIVE 
PRECURSORS 
(REHASH  #4) 


STEP  0  INTERPRETATION 
This  test  checks  the  LRU  ID  resistor, 
and  the  DMM  is  being  used  as  the 
measurement  device.  I  will  initially  focus 
on  the  LRU  because  the  LRU  ID  resistor 
may  actually  be  bad  as  the  failed  test  indicates, 
or  a  pushed  pin  in  the  test  package  could  have 
caused  the  fail.  Since  a  pushed  pin  is  more  likely 
than  a  bad  ID  resistor,  the  problem  is  more  likely 
to  be  in  the  test  package.  I  wont  focus  on  the  test 
station  initially  since  troubleshooting  the  station  is 
more  difficult;  I'll  rule  out  easier  components  first 


LRU-3 

I/F 

ADAPTER 


TEST  PACKAGE  LRU-3 


STEP  1 

A:  Go  to  TO  1 2 P3-2ALR5 6-78-1  to  look  at  test  description 
and  find  out  where  measurements  were  made. 

P:  Need  to  understand  the  failing  test. 

R:  Finds  active  path  is  through  pins  128  and  68  on  J1 2  of  the  LRU. 

I:  This  tells  me  what  I  need  to  know  to  eliminate  the  LRU  ED  resistor. 


LRU-3  ID 
RESISTOR 


TEST  STATION 


STEP  2 

A:  Remove  Pi  from  J12  of  LRU-3,  and  AR1 :  Reads  1999.9  Kohms. 

ohm  out  the  path  between  pins  68  &  128.  AH :  Bad  ID  resistor. 

P:  To  see  if  the  ID  resistor  check  is  good  or  not 
R:  Reading  is  1.55  Mohms. 

I:  The  problem  is  in  the  test  package  or  the  test  station. 


LRU-3  ED 
RESISTOR 


AA1:  Measure  path  at  test 
station  access  panel. 

COSTS/BENEFITS: 

Measuring  at  access  panel  would 
allow  you  to  eliminate  the  test 
package  as  well  as  the  LRU  ID 
resistor.  If  you’rcrurmmg  an 
LRU  when  thefail  occurs,  you 
suspect  the  LRU  first;  measuring 
at  the  access  panel  means  you 
suspect  the  test  equipment 
This  action  is  SLIGHTLY 
WORSE  than  the  original. 


API:  Focus  on  the 
interface  adapter. 
Normally,  IPST 
would  have  checked 
the  I/F  adapter  before 
running  the  LRU. 

But  testing  was 
started  at  segment  10 
of  theLRU  tests.  So 
I  focussed  on  the 
LRU. 

COSTS/BENEFITS: 
MUCH  BETTER  to 


AA2:  Run  shop-standard  LRU.  run  IPST  before  ID 

COSTS/BENEFITS:  resistor  tests. 

MUCH  WORSE  than  original 
action  because  if  s  time  consuming 
to  connect  to  the  shop  standard. 


AA3:  Replace  the  LRU  ID  resistor. 
COSTS/BENEFITS: 

MUCH  WORSE  than  original  action. 
You  should  verify  it’s  bad  before  you 
swap  it  since  replacing  the  resistor  is 
time  consuming. 
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SOLUTION  PATH 


ALTERNATIVE  RESULTS/ 
INTERPRETATIONS 


ALTERNATIVE 

ACTIONS 


ALTERNATIVE 

PRECURSORS 


STEP  3 

A:  Run  EPST  (functional  check)  on  the 
interface  unit 

P:  This  checks  the  internal  circuitry  of 
the  interface  unit  for  this  type  of  test 
Here,  it's  probably  just  a  straight  path. 

R:  All  DC  tests  pass,  all  ohms  tests  fail  open 

Is  This  result  tells  you  the  problem  is  in  the 
test  station  because  ail  the  ohms  checks 
fail;  there  is  nothing  in  the  test  package 
common  to  all  ohms  checks. 


AR1:  All  tests  pass. 

All:  Fault  in  cable  between 
interface  adapter  &  LRU. 

AR2:  DC  tests  fail. 

AR2:  Same  interpretation; 
fault  is  in  test  station. 


AA1:  Manually  check  out  the  test 
package  using  test  package 
schematics  in 
TO  33D7-50-1-151. 

COSTS/BENEFITS; 

MUCH  WORSE  than  original 
action;  requires  much  time,  test 
equipment;  normally  you  would 
use  these  tests  for  troubleshooting. 


STEP  4 

A:  Run  OAFI  on  DMM  and  see 
if  it  passes  the  ohms  checks. 

P:  Since  the  DMM  does  all  ohms  checks, 
need  to  make  sure  it's  working. 

R-  Ohms  checks  are  open,  and  DC 
checks  are  O.K. 

I:  Something  is  wrong  with  the  ohms 
mode  in  the  DMM.  Tm  pretty  sure  that 
everything  up  to  the  instrument  select  is 
good  because  when  we  ran  the  LRU,  we 
used  the  external  test  points,  whereas  the 
OAFI  test  used  internal  test  points;  Since 
different  cards  were  used  in  these  tests, 
the  inputs  to  those  cards  were  probably 
O.K.  The  problem  is  in  the  DMM, 
Instrument  Select,  or  I/O  card  for  the 
DMM  since  those  components  are 
common  to  the  two  failed  tests. 


AR1:  OAFI  on  DMM  passes. 

Alls  OAFI  and  LRU  tests 
could  use  different 
instrument  select  lines. 

In  that  case,  this  result 
would  still  point  to  the 
instrument  select  or  external 
test  points. 


DMM  OHMS-H 


OHMS- 


ENTERFACE 

CHASSIS 


AA1:  Could  use  the  patch 

program  to  hook  DMM 
up  to  a  different  device 
like  the  decade  box  which 
is  a  resistor  network. 

COSTS/BENEFITS: 

The  original  action  is  the 
easiest  way  to  check  out 
the  DMM,  and  since  the 
DMM  is  no  more  suspect 
than  other  components  at 
this  point,  arbitrarily 
hooking  a  device  up  to  the 
front  panel  without  knowing 
all  resistor  readings  would 
fail  is  strange. 


API:  Instrument 
select  lines 
are  possible. 

AP2:  External 
test  points 
possible,  but 
unlikely. 

COSTS/BENEFITS: 
DMM  (original 
precursor)  and 
instrument  select 
lines  are  MUCH 
BETTER  targets 
at  this  point  than 
external  test  points 
given  the  failed 
ohms  tests  and  the 


passed  voltage 
checks. 


STEPS 


A:  Look  up  description  of  OAFI  test, 
in  TO  33D7-38-77-28-1-L 

P:  Need  to  find  out  whafs  involved 

DMM  I/O 

INTERFACE  CHASSIS 

1 

INSTRUMENT 

INTERNAL 

SELF 

in  the  failed  OAFI. 

DMM 

i . 

SELECT 

TEST 

CHECK 

R:  See  drawing  (right). 

J29 

POINTS 

NETWORK 

I:  External  test  points  were  not  used 
in  OAFI  test. 

EXTERNAL  TEST  POINTS 

INTERFACE  ADAPTER 
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SOLUTION  PATH 


ALTERNATIVE  RESULTS/ 
INTERPRETATIONS 


ALTERNATIVE 

ACTIONS 


STEP  6 

A:  Disconnect  DMM  at  J29  to  take  the 
DMM  out  of  the  circuit  Take  an  ohms 
measurement  at  the  access  panel  at 
measurement  bus  (MB)  3  and  MB4. 

Look  for  8  to  12  Kohms. 

P:  If  signal  is  getting  to  the  MB  from  the 
self  check  netwoik,  the  problem  is 
probably  from  the  instrument  select  to  the 
measurement  device.  This  check  will 
ensure  that  both  internal  and  external  test 


AR1 :  Reads  open. 

All:  Reading  wasn't  getting  on 
the  MB;  you  would  have  to 
run  OAFI  on  the  rest  of  the 
interface  chassis.  For  some 
reason,  the  self  check  wasn't 
getting  connected  to  the  MB 
from  internal  test  points. 


points  are  good. 

R:  Reading  within  limits,  +2K  ohms. 

I:  The  circuitry  for  testing  the  DMM  was 
good,  i.e.,  the  MB,  internal  and  external 
test  points,  up  to  the  instrument  select  This 
didn't  rule  out  anything  other  than  what  was 
already  expected  to  be  good.  Still  suspect  the 
DMM  or  the  instrument  select  card.  We  were 
just  getting  a  baseline  ohms  reading  with  this 
measurement 


AA1:  Extend  the  external  test 
points  card  (U20)  and 
measure  its  output 

COSTS/BENEFITS: 

MUCH  BETTER  to  measure 
at  the  access  panel  since  you 
don't  suspect  the  U20  card 
anyway  and  it’s  also  easier. 


FROM 

INTERFACE 

ADAPTER 


MB  ROUTES  SIGNALS  TO  INSTRUMENT  SELECT 


ALTERNATIVE 

PRECURSORS 


STEP  7 

A:  Go  to  TO  33D7-38-77-2-2,  MMX  Functional 
Organization,  and  TO  33D7-38-77-28-1-1, 
DMM  OAFI  test  summary,  to  find  precise 
path  used  in  failed  OAFI  test 
P:  Want  to  know  which  relays  were  being 
used  on  the  instrument  select  card  (U2). 

R:  K9,K13,K18  and  K24  are  used 
I:  Now  I  can  check  continuity  through  the 
instrument  select  (U2)  card  and  if  there's 
an  open.  Til  know  which  relays  to  look  at 

STEP  8 


ACCESS 

PANEL 

TEST 

POINTS 


A:  Ohm  from  MB4  test  point  on  access 
panel  to  connectors  on  J4  and  J5  on  the 
back  of  the  U2  card,  after  disconnecting 
the  internal  test  points  card  (U20)  at  J3 
from  the  self  check  network.  This 
allows  you  to  measure  from  internal 
test  points  to  input  lines  of  DMM 
(through  the  instrument  select  lines). 

P:  Checking  for  opens  in  lines 
going  through  U2  card  to  the 
DMM  ohms  +  and  -  inputs. 

R:  IK  ohms  through  each  connector. 

I:  The  relays  are  setting  as  they  should 
There  might  still  be  something  wrong 
with  the  card  such  as  a  relay  being  set 
that's  not  supposed  to  be. 


AR1:  Open 

All:  There  could  be  a  bad  relay 
on  the  U2  card,  or  the  logic 
to  one  of  the  relays  is  bad 


AA1:  Could  have  checked  the 
path  by  probing  with  the  U2 
extended  and  measuring  from 
the  input  to  the  output  of  the  card 
COSTS/BENEFITS: 

This  action  is  SLIGHTLY 
WORSE  than  the  original 
because  there  is  no  advantage 
to  it,  and  you  can  damage  the 
card  by  handling  it 
AA2:  With  card  extended 
measure  across  the  relay 
input  to  output 
COSTS/BENEFITS: 

This  has  additional  disadvantage 
(besides  handling  card)  that  the 
runs  (paths)  on  the  card  are  not 
being  checked 


STEPS  8 -11 

The  main  goal  of 
the  original  action 
was  to  check  the 
instrument  select 
relays.  In  looking 
at  the  total  circuit, 
you  could  have 
ohmed  out  any¬ 
where  you  could 
break  it  down. 
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SOLUTION  PATH 

ALTERNATIVE  RESULTS/  ALTERNATIVE 

INTERPRETATIONS  ACTIONS 

ALTERNATIVE 

PRECURSORS 

STEP  9 

AR1:  Good  continuity 

API:  Look  for  shorted 

A: 

Ohm  check  from  J4  and  J5 

through  both  lines. 

relay  on  instrument 

(outputs  of  U2  card)  to  J29  on 
die  DMM,  pins  9  and  7. 

All :  Could  have  a  shorted 
relay  on  the  U2  card. 

select  cards. 

P: 

Checking  continuity  through 
the  CMG  back  to  the  DMM. 

AR2:  Could  have  found  good 

continuity  through  the  high 

R: 

When  you  disconnect  the  cable 
from  J5,  you  find  the  center 

line  and  an  open  in  the  low  line. 

A12:  Problem  is  in  the  low  line 

Is 

conductor  is  pushed  in. 

Problem  solved. 

(OHMS  -)tothe  DMM 

GROUPED  ACTIONS 
(REHASH  #5) 


ACTIONS 

1.  Look  at  LRU  TO  to  understand  failing  test 

2.  Ohm  out  LRU  ID  resistor. 

3.  Run  IPST  on  interface  adapter. 

4.  Run  OAFI  on  DMM. 

5.  Look  at  OAFI  TO  to  understand  failed  test 

6.  Disconnect  DMM,  take  ohms  meas.  at  access  panel. 

7.  Look  at  block  diagrams  TO  to  identity  instrument  select 
relays  being  used  in  failed  OAFI  test 

8.  Ohms  check:  from  access  panel  through  instrument  select  relays. 

9.  Ohms  check  from  output  of  U2  to  the  input  of  the  DMM. 


FIRST  GROUPING  SECOND  GROUPING 

1  &  2:Ruling  out  the  LRU  1-3:  Eliminate  components 

3:  Ruling  out  the  test  package  external  to  test  station. 

4  &  5:  Correlating  results  of  multiple  6-9:  Eliminate  internal  test 
tests.  station  components. 

6:  Space  splitting;  confirming 

interpretation  of  multiple  test  results. 

7  &  8:  Checking  instrument  select  relays. 

9:  Checking  ohms  path  to  the  DMM 


no 


Appendix  E 


Problem  Set  Review  by  Expert  Problem  Solvers 

As  a  set,  how  representative  of  the  problems  seen  in  the  shop  are  these  problems? 

5  4  3  2  1 

highly  moderately  not 

representative  representative  representative 

Do  you  think  these  problems  cover  all  of  the  important  thinking  skills  in  the  job?  (circle  one) 

Yes  No 

If  not,  what  other  thinking  skills  are  important  in  this  job? 

What  types  of  problems  would  tap  these  additional  skills? 


How  would  you  rate  the  proficiency  of  a  technician  who  can  solve  these  problems? 

5  4  3  2  1 

highly  moderately  not 

proficient  proficient  proficient 

If  you  rated  the  proficiency  3, 2,  or  1,  briefly  describe  what  other  skills  technicians  must  posses  or 
other  tasks  technicians  must  be  able  to  accomplish  for  you  to  rate  them  above  moderately  proficient? 


How  useful  are  these  problems  for  training  technicians? 

5  4  3  2  1 

very  moderately  not 

useful  useful  useful 

If  you  rated  the  usefulness  3, 2,  or  1,  briefly  explain  why  the  problems  are  not  more  useful. 


Please  write  any  additional  comments  you  wish  to  make  about  the  problem  set. 


Ill 


Scales  for  Skill  Criticality  Ratings 


Usefulness  of  Skill 

5  =  Extremely  useful 

4  =  Very  useful 

3  =  Moderately  useful 

2  =  Slightly  useful 

1  =  Not  at  all  useful 

Learning  Difficulty 

5  =  Very  difficult  to  learn 

4  =  Difficult  to  learn 

3  =  Moderately  difficult  to  learn 

2  =  Easy  to  learn 

1  =  Very  easy  to  learn 

Recommended  Training  Emphasis 

5  =  Extremely  important  to  train 

4  =  Very  important  to  train 

3  =  Moderately  important  to  train 

2  =  Slightly  important  to  train 

1  =  Not  at  all  important  to  train 


112 


Problem  Difficulty  Rankings 


Rank  the  problems  in  order  of  difficulty,  with  the  easiest  problem  as  number  1.  Write  the  difficulty 
rank  for  experts  in  the  first  column.  Write  the  difficulty  rank  for  first-termers  in  the  second  column. 

Expert  1  st-T  ermer 

1.  A/D  Switching/Routing  _  _ 

2.  R/F  Switching/Routing  _  _ 

3.  RF  Waveform  Analysis  _  _ 

4.  Analog  Waveform  Analysis  _  _ 

5.  Data  Transmission  (I/O  Logic)  _  _ 

6.  Digital  Data  Analysis  _  _ 

7.  Clock/Timing  _  _ 

8.  Load  _  _ 

9.  Power  Supply  _  _ 

10.  Computer  Error  Messages  _  _ 

11.  Software  Maintenance  _  _ 

12.  Connectors  _  _ 

13.  Initial  Equipment  Setup  _  _ 

14.  Computer  Hardware  _  _ 

15.  Alignments  _  _ 
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Appendix  F 

Independent  (Advanced)  Expert  Problem  Set  Review 


Name 

Rank 

AFSC 

Base 


As  a  set,  how  representative  of  the  problems  seen  in  the  shop  are  these  problems? 

5  4  3  2  1 

highly  moderately  not 

representative  representative  representative 

Do  you  think  these  problems  cover  all  of  the  important  cognitive  skills  in  the  job?  (circle  one) 

Yes  No 

If  not,  what  other  thinking  skills  are  important  in  this  job? 


What  types  of  problems  would  tap  these  additional  skills? 


How  would  you  rate  the  proficiency  of  a  technician  who  can  solve  these  problems? 

5  4  3  2  1 

highly  moderately  not 

proficient  proficient  proficient 

If  you  rated  the  proficiency  3,  2,  or  1,  briefly  describe  what  other  skills  technicians  must  posses  or 
other  tasks  technicians  must  be  able  to  accomplish  for  you  to  rate  them  above  moderately  proficient? 


How  useful  are  these  problems  for  training  technicians? 

5  4  3  2 

very  moderately 

useful  useful 


1 

not 

useful 


If  you  rated  the  usefulness  3,  2,  or  1,  briefly  explain  why  the  problems  are  not  more  useful. 


Please  write  any  additional  comments  you  wish  to  make  about  the  problem  set. 
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Is  each  problem  summary  accurate?  If  not,  please  rewrite  the  portion  of  the  summary  that  needs 
correction. 

Circle  One 


1.  A/D  Switching/Routing . 

. OK  as  is . 

. See  correction  on  summary 

2.  RF  Switching/Routing . 

. OK  as  is . 

. See  correction  on  summary 

3.  RF  Waveform  Analysis . 

. OK  as  is . 

. See  correction  on  summary 

4.  Analog  Waveform  Analysis . 

. OK  as  is . 

. See  correction  on  summary 

5.  Data  Transmission  (I/O  Logic) . 

. OK  as  is . 

. See  correction  on  summary 

6.  Digital  Data  Analysis . 

. OK  as  is . 

. See  correction  on  summary 

7.  Clock  Timing  . 

. OK  as  is . 

. See  correction  on  summary 

8.  Load  . 

. OK  as  is . 

. See  correction  on  summary 

9.  Power  Supply . 

. OK  as  is . 

. See  correction  on  summary 

10.  Computer  Error  Messages . 

. OK  as  is . 

. See  correction  on  summary 

11.  Software  Maintenance . 

. OK  as  is . 

. See  correction  on  summary 

12.  Connectors . 

. OK  as  is . 

. See  correction  on  summary 

13.  Initial  Equipment  Setup . 

. OK  as  is . 

. See  correction  on  summary 

14.  Computer  Hardware . 

. OK  as  is. . 

. See  correction  on  summary 

15.  Alignments . 

. OK  as  is . 

. See  correction  on  summary 
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Are  any  of  these  problems  things  that  technicians  in  this  job  at  other  sites  would  not  see? 
Yes  No 

If  yes,  briefly  note  next  to  the  problem  category  what  is  site-specific  about  the  problem. 

1.  A/D  Switching/Routing: 

2.  RF  Switching/Routing: 

3.  RF  Waveform  Analysis: 

4.  Analog  Waveform  Analysis: 

5.  Data  Transmission: 

6.  Digital  Data  Analysis: 

7.  Clock/Timing: 

8.  Load: 

9.  Power  Supply: 

10.  Computer  Error  Messages: 

11.  Software  Maintenance: 

12.  Connectors: 

13.  Initial  Equipment  Setup: 

14.  Computer  Hardware: 

15.  Alignments: 
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Are  any  of  these  problems  types  of  things  that  would  only  occur  on  the  specific  equipment  you  have  at 
this  site? 

Yes  No 

If  yes,  briefly  note  next  to  the  problem  category  what  is  equipment-specific  about  the  problem. 

1.  A/D  Switching  Routing: 

2.  RF  Switching/Routing: 

3.  RF  Waveform  Analysis: 

4.  Analog  Waveform  Analysis: 

5.  Data  Transmission: 

6.  Digital  Data  Analysis: 

7.  Clock/Timing: 

8.  Load: 

9.  Power  Supply: 

10.  Computer  Error  Messages: 


11.  Software  Maintenance: 


12.  Connectors: 


13.  Initial  Equipment  Setup: 


14.  Computer  Hardware: 

15.  Alignments: 
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Would  any  of  these  problems  be  solved  differently  at  other  sites  than  they  are  here?  (For  example, 
would  procedures  such  as  swapping  be  used  more  or  less  frequently?) 

Yes  No 

If  yes,  write  the  type  of  site  where  the  solution  would  differ  next  to  the  number  of  the  problem,  and 
briefly  describe  how  the  solution  would  be  different  at  these  other  sites. 

1.  A/D  Switching  Routing: 

2.  RF  Switching/Routing: 

3.  RF  Waveform  Analysis: 

4.  Analog  Waveform  Analysis: 

5.  Data  Transmission: 

6.  Digital  Data  Analysis: 

7.  Clock/Timing: 

8.  Load: 

9.  Power  Supply: 

10.  Computer  Error  Messages: 

11.  Software  Maintenance: 

12.  Connectors: 

13.  Initial  Equipment  Setup: 

14.  Computer  Hardware: 

15.  Alignments: 
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